[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:23 dist-test-slave-h6n0 krb5kdc[12739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:23 dist-test-slave-h6n0 krb5kdc[12739](info): set up 2 sockets
Aug 12 11:23:23 dist-test-slave-h6n0 krb5kdc[12739](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260812 11:23:24.360872 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.415s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:23:24 dist-test-slave-h6n0 krb5kdc[12739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:23:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:23:24.603036 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:33831
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:24.976346 12760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:24.976653 12760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:24.976783 12760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:24.984048 12760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:23:24.984156 12760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:24.984205 12760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:24.984256 12760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:23:24.984294 12760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:23:24.994165 12760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:33831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:24.995683 12760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:24.997056 12760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:25.011981 12770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:25.012051 12775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:25.011947 12771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:25.012979 12760 server_base.cc:1034] running on GCE node
I20260812 11:23:25.014210 12760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:25.016412 12760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:25.017705 12760 hybrid_clock.cc:648] HybridClock initialized: now 1786533805017624 us; error 79 us; skew 500 ppm
I20260812 11:23:25.018188 12760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:25 dist-test-slave-h6n0 krb5kdc[12739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533805, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:25.022701 12760 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:23:25.024490 12760 webserver.cc:460] Webserver started at http://127.12.110.254:39001/ using document root <none> and password file <none>
I20260812 11:23:25.025225 12760 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:25.025342 12760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:25.025626 12760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:25.029999 12760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "2247f227de784a09a68d10ab58e4c398"
format_stamp: "Formatted at 2026-08-12 11:23:25 on dist-test-slave-h6n0"
server_key: "926ac894bd47bfac68706e253ebc222f"
server_key_iv: "52636b656a8943ce759c28e1b32d770d"
server_key_version: "encryptionkey@0"
I20260812 11:23:25.031271 12760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "2247f227de784a09a68d10ab58e4c398"
format_stamp: "Formatted at 2026-08-12 11:23:25 on dist-test-slave-h6n0"
server_key: "926ac894bd47bfac68706e253ebc222f"
server_key_iv: "52636b656a8943ce759c28e1b32d770d"
server_key_version: "encryptionkey@0"
I20260812 11:23:25.038926 12760 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260812 11:23:25.043622 12787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:25.045331 12760 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:23:25.045493 12760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "2247f227de784a09a68d10ab58e4c398"
format_stamp: "Formatted at 2026-08-12 11:23:25 on dist-test-slave-h6n0"
server_key: "926ac894bd47bfac68706e253ebc222f"
server_key_iv: "52636b656a8943ce759c28e1b32d770d"
server_key_version: "encryptionkey@0"
I20260812 11:23:25.045699 12760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:25.079600 12760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:25.084436 12760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:25.084717 12760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:25.107085 12760 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:33831
I20260812 11:23:25.107095 12883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:33831 every 8 connection(s)
I20260812 11:23:25.108582 12760 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:23:25.112349 12884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:25.118070 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 12760
I20260812 11:23:25.118453 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:23:25.121037 12731 external_mini_cluster.cc:1428] Setting key b840e2be976d9586425a440f14960805
I20260812 11:23:25.126206 12884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398: Bootstrap starting.
I20260812 11:23:25.129907 12884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:25.131021 12884 log.cc:824] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:25.133612 12884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398: No bootstrap required, opened a new log
I20260812 11:23:25.141975 12884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2247f227de784a09a68d10ab58e4c398" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 33831 } }
I20260812 11:23:25.142356 12884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:25.142499 12884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2247f227de784a09a68d10ab58e4c398, State: Initialized, Role: FOLLOWER
I20260812 11:23:25.143458 12884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2247f227de784a09a68d10ab58e4c398" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 33831 } }
I20260812 11:23:25.143968 12884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:23:25.144374 12884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:23:25.144735 12884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:25.147724 12884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2247f227de784a09a68d10ab58e4c398" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 33831 } }
I20260812 11:23:25.148285 12884 leader_election.cc:302] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2247f227de784a09a68d10ab58e4c398; no voters:
I20260812 11:23:25.148949 12884 leader_election.cc:288] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:23:25.149106 12898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:25.151361 12884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:23:25.152664 12898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [term 1 LEADER]: Becoming Leader. State: Replica: 2247f227de784a09a68d10ab58e4c398, State: Running, Role: LEADER
I20260812 11:23:25.154462 12898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2247f227de784a09a68d10ab58e4c398" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 33831 } }
I20260812 11:23:25.162179 12899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2247f227de784a09a68d10ab58e4c398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2247f227de784a09a68d10ab58e4c398" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 33831 } } }
Aug 12 11:23:25 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:25.162653 12899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:25.163177 12901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2247f227de784a09a68d10ab58e4c398. Latest consensus state: current_term: 1 leader_uuid: "2247f227de784a09a68d10ab58e4c398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2247f227de784a09a68d10ab58e4c398" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 33831 } } }
I20260812 11:23:25.163501 12901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:25.174381 12914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:23:25.182822 12914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:23:25.182880 12891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:25.126162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.12.110.254:33831)
0812 11:23:25.128187 (+ 2025us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:25.128201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:25.128273 (+ 72us) server_negotiation.cc:407] Connection header received
0812 11:23:25.141504 (+ 13231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:25.141528 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:25.141927 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:25.142364 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:25.146610 (+ 4246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:25.148044 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:25.151039 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:25.154333 (+ 3294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:25.154617 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:25.166326 (+ 11709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:25.166346 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:25.166381 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:25.166487 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:25.173442 (+ 6955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:25.175185 (+ 1743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:25.175199 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:25.175210 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:25.175542 (+ 332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:25.176282 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:25.176291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:25.176297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:25.177412 (+ 1115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:25.177850 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:25.180587 (+ 2737us) server_negotiation.cc:299] Negotiation successful
0812 11:23:25.181189 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1731,"thread_start_us":1492,"threads_started":1}
I20260812 11:23:25.193212 12914 catalog_manager.cc:1347] Generated new cluster ID: 7bfe006f734d41729b22228a1eb7f75c
I20260812 11:23:25.193354 12914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:23:25.215380 12914 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:23:25.216888 12914 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:23:25.227802 12914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2247f227de784a09a68d10ab58e4c398: Generated new TSK 0
I20260812 11:23:25.228997 12914 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:23:25.303122 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:33831
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:25.714198 12931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:25.714504 12931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:25.714701 12931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:25.722195 12931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:25.722314 12931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:25.722497 12931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:23:25.732511 12931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:33831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:25.733881 12931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:25.735332 12931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:25.750881 12937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:25.752341 12939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:25.755843 12936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:25.757675 12931 server_base.cc:1034] running on GCE node
I20260812 11:23:25.758401 12931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:25.759500 12931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:25.760870 12931 hybrid_clock.cc:648] HybridClock initialized: now 1786533805760765 us; error 70 us; skew 500 ppm
I20260812 11:23:25.761516 12931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:25 dist-test-slave-h6n0 krb5kdc[12739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533805, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:25.765338 12931 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:23:25.767117 12931 webserver.cc:460] Webserver started at http://127.12.110.193:38253/ using document root <none> and password file <none>
I20260812 11:23:25.767829 12931 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:25.767940 12931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:25.768234 12931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:25.771320 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "a8ae45bfe50745df99c41cea65e63bfb"
format_stamp: "Formatted at 2026-08-12 11:23:25 on dist-test-slave-h6n0"
server_key: "fe660a014ef72c66be81114468fe21f3"
server_key_iv: "505d321eb973a784383c038923b4f009"
server_key_version: "encryptionkey@0"
I20260812 11:23:25.772063 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "a8ae45bfe50745df99c41cea65e63bfb"
format_stamp: "Formatted at 2026-08-12 11:23:25 on dist-test-slave-h6n0"
server_key: "fe660a014ef72c66be81114468fe21f3"
server_key_iv: "505d321eb973a784383c038923b4f009"
server_key_version: "encryptionkey@0"
I20260812 11:23:25.777365 12931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:23:25.781273 12951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:25.783627 12931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 11:23:25.783852 12931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "a8ae45bfe50745df99c41cea65e63bfb"
format_stamp: "Formatted at 2026-08-12 11:23:25 on dist-test-slave-h6n0"
server_key: "fe660a014ef72c66be81114468fe21f3"
server_key_iv: "505d321eb973a784383c038923b4f009"
server_key_version: "encryptionkey@0"
I20260812 11:23:25.784080 12931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:25.814433 12931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:25.819985 12931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:25.820560 12931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:25.821952 12931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:25.824241 12931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:25.824347 12931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:25.824458 12931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:25.824509 12931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:25.864494 12931 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:45387
I20260812 11:23:25.864600 13143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:45387 every 8 connection(s)
I20260812 11:23:25.866397 12931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:23:25.867499 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 12931
I20260812 11:23:25.867841 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:23:25.868732 12731 external_mini_cluster.cc:1428] Setting key d44c202b64dd064c94ab3b6e42d40bd9
Aug 12 11:23:25 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533805, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:25.891368 13146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:25.870567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:41271 (local address 127.12.110.254:33831)
0812 11:23:25.871292 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:25.871299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:25.872792 (+ 1493us) server_negotiation.cc:407] Connection header received
0812 11:23:25.874855 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:25.874860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:25.874953 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:25.875078 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:25.876699 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:25.877417 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:25.879069 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:25.879666 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:25.879935 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:25.884963 (+ 5028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:25.884969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:25.884973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:25.885020 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:25.887930 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:25.888895 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:25.888902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:25.888905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:25.889052 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:25.889554 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:25.889558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:25.889560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:25.889831 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:25.890007 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:25.890747 (+ 740us) server_negotiation.cc:299] Negotiation successful
0812 11:23:25.890984 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":492,"threads_started":1}
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
I20260812 11:23:25.892434 13149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:25.871500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:33831 (local address 127.12.110.193:41271)
0812 11:23:25.872617 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:25.872671 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:25.874520 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:25.875329 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:25.875337 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:25.875898 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:25.876516 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:25.876533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:25.877599 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:25.877603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:25.878916 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:25.878928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:25.880137 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:25.880150 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:25.880524 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:25.881549 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:25.881579 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:25.884594 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:25.888123 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:25.888134 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:25.888150 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:25.888760 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:25.889171 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:25.889176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:25.889180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:25.889384 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:25.889975 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:25.889983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:25.890326 (+ 343us) client_negotiation.cc:769] Sending connection context
0812 11:23:25.890634 (+ 308us) client_negotiation.cc:240] Negotiation successful
0812 11:23:25.891194 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":549,"threads_started":1}
I20260812 11:23:25.894852 13144 heartbeater.cc:347] Connected to a master server at 127.12.110.254:33831
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
I20260812 11:23:25.895246 13144 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:25.896222 13144 heartbeater.cc:511] Master 127.12.110.254:33831 requested a full tablet report, sending...
I20260812 11:23:25.899461 12820 ts_manager.cc:194] Registered new tserver with Master: a8ae45bfe50745df99c41cea65e63bfb (127.12.110.193:45387)
I20260812 11:23:25.900988 12820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:41271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:23:25.923081 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:33831
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:26.312098 13153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:26.312393 13153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:26.312493 13153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:26.320052 13153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:26.320184 13153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:26.320302 13153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:23:26.331825 13153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:33831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:26.333591 13153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:26.335464 13153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:26.351018 13161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:26.350373 13160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:26.351307 13165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:26.352579 13153 server_base.cc:1034] running on GCE node
I20260812 11:23:26.353302 13153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:26.354700 13153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:26.356114 13153 hybrid_clock.cc:648] HybridClock initialized: now 1786533806355958 us; error 30 us; skew 500 ppm
I20260812 11:23:26.356613 13153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:26 dist-test-slave-h6n0 krb5kdc[12739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533806, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:26.360426 13153 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:23:26.361958 13153 webserver.cc:460] Webserver started at http://127.12.110.194:36641/ using document root <none> and password file <none>
I20260812 11:23:26.362752 13153 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:26.362865 13153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:26.363196 13153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:26.366151 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "de6a108e88c1471ca4ca4c3a908f09f2"
format_stamp: "Formatted at 2026-08-12 11:23:26 on dist-test-slave-h6n0"
server_key: "ef38dc460b0283818b239ef008899854"
server_key_iv: "48e71f10de2ea989f8ea8267b8905c78"
server_key_version: "encryptionkey@0"
I20260812 11:23:26.366928 13153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "de6a108e88c1471ca4ca4c3a908f09f2"
format_stamp: "Formatted at 2026-08-12 11:23:26 on dist-test-slave-h6n0"
server_key: "ef38dc460b0283818b239ef008899854"
server_key_iv: "48e71f10de2ea989f8ea8267b8905c78"
server_key_version: "encryptionkey@0"
I20260812 11:23:26.372781 13153 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:23:26.376305 13179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:26.377652 13153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:23:26.377835 13153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "de6a108e88c1471ca4ca4c3a908f09f2"
format_stamp: "Formatted at 2026-08-12 11:23:26 on dist-test-slave-h6n0"
server_key: "ef38dc460b0283818b239ef008899854"
server_key_iv: "48e71f10de2ea989f8ea8267b8905c78"
server_key_version: "encryptionkey@0"
I20260812 11:23:26.378002 13153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:26.395972 13153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:26.400346 13153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:26.400651 13153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:26.401679 13153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:26.403170 13153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:26.403250 13153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:26.403393 13153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:26.403445 13153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:26.445505 13153 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:34889
I20260812 11:23:26.445547 13368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:34889 every 8 connection(s)
I20260812 11:23:26.447541 13153 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:23:26.449266 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13153
I20260812 11:23:26.449506 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:23:26.450120 12731 external_mini_cluster.cc:1428] Setting key c512f66c2128a9aba109b4da22a3b27e
Aug 12 11:23:26 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533806, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
I20260812 11:23:26.474920 13373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:26.450841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:33685 (local address 127.12.110.254:33831)
0812 11:23:26.451458 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:26.451463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:26.452876 (+ 1413us) server_negotiation.cc:407] Connection header received
0812 11:23:26.456173 (+ 3297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:26.456180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:26.456409 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:26.456580 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:26.458894 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:26.459799 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:26.462197 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:26.462760 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:26.462907 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:26.466881 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:26.466886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:26.466889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:26.466923 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:26.470841 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:26.471990 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:26.471999 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:26.472008 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:26.472165 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:26.472688 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:26.472698 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:26.472700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:26.473025 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:26.473210 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:26.474119 (+ 909us) server_negotiation.cc:299] Negotiation successful
0812 11:23:26.474423 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":288,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
I20260812 11:23:26.476209 13374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:26.451548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:33831 (local address 127.12.110.194:33685)
0812 11:23:26.452692 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:26.452741 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:26.455702 (+ 2961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:26.456881 (+ 1179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:26.456895 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:26.457770 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:26.458620 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:26.458647 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:26.460008 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:26.460016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:26.462019 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:26.462032 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:26.463016 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:26.463025 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:26.463260 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:26.464232 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:26.464260 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:26.466626 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:26.471121 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:26.471133 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:26.471151 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:26.471729 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:26.472300 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:26.472306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:26.472309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:26.472549 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:26.473173 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:26.473182 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:26.473684 (+ 502us) client_negotiation.cc:769] Sending connection context
0812 11:23:26.474004 (+ 320us) client_negotiation.cc:240] Negotiation successful
0812 11:23:26.474726 (+ 722us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":509,"threads_started":1}
I20260812 11:23:26.478420 13370 heartbeater.cc:347] Connected to a master server at 127.12.110.254:33831
I20260812 11:23:26.478820 13370 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:26.480126 13370 heartbeater.cc:511] Master 127.12.110.254:33831 requested a full tablet report, sending...
I20260812 11:23:26.482501 12820 ts_manager.cc:194] Registered new tserver with Master: de6a108e88c1471ca4ca4c3a908f09f2 (127.12.110.194:34889)
I20260812 11:23:26.483592 12820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:33685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:23:26.507447 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:33831
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:26.884845 13377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:26.885432 13377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:26.885612 13377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:26.895197 13377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:26.895345 13377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:26.895480 13377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:23:26.904464 13144 heartbeater.cc:503] Master 127.12.110.254:33831 was elected leader, sending a full tablet report...
I20260812 11:23:26.905692 13377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:33831
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:26.907369 13377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:26.908686 13377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:26.923470 13387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:26.923494 13388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:26.923839 13395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:26.925416 13377 server_base.cc:1034] running on GCE node
I20260812 11:23:26.926179 13377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:26.927309 13377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:26.928570 13377 hybrid_clock.cc:648] HybridClock initialized: now 1786533806928515 us; error 48 us; skew 500 ppm
I20260812 11:23:26.928936 13377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:26 dist-test-slave-h6n0 krb5kdc[12739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533806, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:26.932451 13377 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:23:26.934170 13377 webserver.cc:460] Webserver started at http://127.12.110.195:33077/ using document root <none> and password file <none>
I20260812 11:23:26.934996 13377 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:26.935151 13377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:26.935493 13377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:26.938776 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "1c5360005b5549318cbf60570ac9c4df"
format_stamp: "Formatted at 2026-08-12 11:23:26 on dist-test-slave-h6n0"
server_key: "bc35807e596338a0f612b093c10312ed"
server_key_iv: "f559b4628dc8bcbcbbdb92cdf628e0ec"
server_key_version: "encryptionkey@0"
I20260812 11:23:26.939625 13377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "1c5360005b5549318cbf60570ac9c4df"
format_stamp: "Formatted at 2026-08-12 11:23:26 on dist-test-slave-h6n0"
server_key: "bc35807e596338a0f612b093c10312ed"
server_key_iv: "f559b4628dc8bcbcbbdb92cdf628e0ec"
server_key_version: "encryptionkey@0"
I20260812 11:23:26.946022 13377 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 11:23:26.950260 13404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:26.951946 13377 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:23:26.952106 13377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "1c5360005b5549318cbf60570ac9c4df"
format_stamp: "Formatted at 2026-08-12 11:23:26 on dist-test-slave-h6n0"
server_key: "bc35807e596338a0f612b093c10312ed"
server_key_iv: "f559b4628dc8bcbcbbdb92cdf628e0ec"
server_key_version: "encryptionkey@0"
I20260812 11:23:26.952359 13377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:26.966985 13377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:26.971462 13377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:26.971786 13377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:26.973120 13377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:26.974691 13377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:26.974798 13377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:26.974964 13377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:26.975112 13377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:27.014380 13377 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:43789
I20260812 11:23:27.014413 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:43789 every 8 connection(s)
I20260812 11:23:27.015874 13377 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:23:27.024557 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13377
I20260812 11:23:27.024868 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:23:27.025468 12731 external_mini_cluster.cc:1428] Setting key 961faa547349128adc389ab9eb2938c7
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533806, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:27.044960 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:27.019789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:48985 (local address 127.12.110.254:33831)
0812 11:23:27.020452 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:27.020457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:27.021780 (+ 1323us) server_negotiation.cc:407] Connection header received
0812 11:23:27.024266 (+ 2486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:27.024278 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:27.024413 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:27.024594 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:27.026973 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.027953 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.030174 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.030988 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.031194 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:27.035739 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:27.035745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:27.035749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:27.035790 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:27.039388 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.040450 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.040455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.040457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.040632 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.041675 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.041682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.041686 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.042238 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:27.042661 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:27.044095 (+ 1434us) server_negotiation.cc:299] Negotiation successful
0812 11:23:27.044407 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":364,"threads_started":1}
I20260812 11:23:27.046442 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:27.020494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:33831 (local address 127.12.110.195:48985)
0812 11:23:27.021580 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:27.021633 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:27.023763 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:27.025182 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:27.025194 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:27.025940 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:27.026680 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:27.026703 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.028175 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.028182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:27.029900 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:27.029914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.031273 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.031278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:27.031479 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:27.032292 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:27.032333 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:27.035402 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:27.039757 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:27.039769 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:27.039785 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:27.040279 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:27.040853 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:27.040858 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:27.040902 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:27.041368 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:27.042777 (+ 1409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:27.042792 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:27.043513 (+ 721us) client_negotiation.cc:769] Sending connection context
0812 11:23:27.043893 (+ 380us) client_negotiation.cc:240] Negotiation successful
0812 11:23:27.044788 (+ 895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":416,"threads_started":1}
I20260812 11:23:27.048489 13591 heartbeater.cc:347] Connected to a master server at 127.12.110.254:33831
I20260812 11:23:27.048923 13591 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:27.049993 13591 heartbeater.cc:511] Master 127.12.110.254:33831 requested a full tablet report, sending...
I20260812 11:23:27.052238 12820 ts_manager.cc:194] Registered new tserver with Master: 1c5360005b5549318cbf60570ac9c4df (127.12.110.195:43789)
I20260812 11:23:27.053138 12820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:48985
I20260812 11:23:27.059067 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[12739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533807, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:27.123395 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:27.101307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.12.110.193:45387)
0812 11:23:27.102248 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:27.102255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:27.102282 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:23:27.102345 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:27.102351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:27.102642 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:27.102774 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:27.104922 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.106251 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.108430 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.109060 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.109179 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:27.115196 (+ 6017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:27.115225 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:27.115247 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:27.115311 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:27.119558 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.120659 (+ 1101us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.120668 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.120671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.120892 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.121668 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.121673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.121675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.122096 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:27.122374 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:27.122751 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 11:23:27.123005 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":662,"threads_started":1}
W20260812 11:23:27.124863 12998 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53744
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[12739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:27.187337 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:27.168601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.12.110.193:45387)
0812 11:23:27.168871 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:27.168880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:27.169237 (+ 357us) server_negotiation.cc:407] Connection header received
0812 11:23:27.169466 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:27.169470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:27.169578 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:27.169697 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:27.171230 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.171933 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.173181 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.173623 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.173757 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:27.181417 (+ 7660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:27.181423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:27.181427 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:27.181460 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:27.184324 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.184994 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.184999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.185002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.185144 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.185901 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.185906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.185908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.186185 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:27.186403 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:27.186902 (+ 499us) server_negotiation.cc:299] Negotiation successful
0812 11:23:27.187090 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 11:23:27.190011 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 12931
I20260812 11:23:27.207844 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:27.191525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53764 (local address 127.12.110.193:45387)
0812 11:23:27.191791 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:27.191796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:27.192122 (+ 326us) server_negotiation.cc:407] Connection header received
0812 11:23:27.192309 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:27.192314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:27.192385 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:27.192475 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:27.193704 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.194602 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.196348 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.196771 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.196887 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:27.200449 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:27.200456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:27.200461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:27.200509 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:27.204878 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.205781 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.205786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.205788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.205971 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.206385 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.206391 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.206393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.206847 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:27.207074 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:27.207365 (+ 291us) server_negotiation.cc:299] Negotiation successful
0812 11:23:27.207567 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 11:23:27.208725 13137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:27.297365 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 12931
I20260812 11:23:27.330847 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13153
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:23:27.350682 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:27.331729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.12.110.194:34889)
0812 11:23:27.332296 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:27.332304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:27.332330 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:23:27.332394 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:27.332400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:27.332599 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:27.332729 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:27.335071 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.336079 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.337766 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.338636 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.338786 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:27.343384 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:27.343397 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:27.343412 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:27.343449 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:27.347505 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.348299 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.348307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.348310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.348494 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.348902 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.348906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.348908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.349285 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:27.349702 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:27.349953 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 11:23:27.350196 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":280,"threads_started":1}
I20260812 11:23:27.352100 13363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:27.444315 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13153
I20260812 11:23:27.474488 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13377
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[12739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:23:27.493846 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:27.475801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.12.110.195:43789)
0812 11:23:27.476350 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:27.476357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:27.476387 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:23:27.476448 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:27.476454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:27.476678 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:27.476855 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:27.478136 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.479243 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.480537 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:27.481072 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:27.481226 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:27.486861 (+ 5635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:27.486874 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:27.486889 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:27.486926 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:27.490245 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.491149 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.491157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.491159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.491305 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:27.491928 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:27.491932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:27.491935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:27.492335 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:27.492605 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:27.493049 (+ 444us) server_negotiation.cc:299] Negotiation successful
0812 11:23:27.493423 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":253,"threads_started":1}
I20260812 11:23:27.496583 13584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:27.598054 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13377
I20260812 11:23:27.629365 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 12760
I20260812 11:23:27.630803 12877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:27.744617 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 12760
2026-08-12T11:23:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4866 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[13630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[13630](info): set up 2 sockets
Aug 12 11:23:27 dist-test-slave-h6n0 krb5kdc[13630](info): commencing operation
krb5kdc: starting...
W20260812 11:23:29.165194 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.363s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:23:29 dist-test-slave-h6n0 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:23:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:23:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:23:29.416375 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37699
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:29.777330 13652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:29.777623 13652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:29.777805 13652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:29.785395 13652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:23:29.785502 13652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:29.785611 13652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:29.785719 13652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:23:29.785799 13652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:23:29.796501 13652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:29.797952 13652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:29.799247 13652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:29.812014 13666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:29.812016 13662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:29.812307 13670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:29.813066 13652 server_base.cc:1034] running on GCE node
I20260812 11:23:29.814391 13652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:29.816493 13652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:29.817835 13652 hybrid_clock.cc:648] HybridClock initialized: now 1786533809817794 us; error 87 us; skew 500 ppm
I20260812 11:23:29.818365 13652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:29 dist-test-slave-h6n0 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533809, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:29.822429 13652 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:23:29.824347 13652 webserver.cc:460] Webserver started at http://127.12.110.254:35473/ using document root <none> and password file <none>
I20260812 11:23:29.825210 13652 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:29.825381 13652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:29.825729 13652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:29.828883 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "eef94a8c4b734efea35efd713eabbba7"
format_stamp: "Formatted at 2026-08-12 11:23:29 on dist-test-slave-h6n0"
server_key: "ec3ecde72560a9fed3b1b83901bdb06d"
server_key_iv: "0eb17d062ba5c60735ef20fc2043aeb0"
server_key_version: "encryptionkey@0"
I20260812 11:23:29.829747 13652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "eef94a8c4b734efea35efd713eabbba7"
format_stamp: "Formatted at 2026-08-12 11:23:29 on dist-test-slave-h6n0"
server_key: "ec3ecde72560a9fed3b1b83901bdb06d"
server_key_iv: "0eb17d062ba5c60735ef20fc2043aeb0"
server_key_version: "encryptionkey@0"
I20260812 11:23:29.835544 13652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:23:29.840426 13684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:29.842123 13652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260812 11:23:29.842392 13652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "eef94a8c4b734efea35efd713eabbba7"
format_stamp: "Formatted at 2026-08-12 11:23:29 on dist-test-slave-h6n0"
server_key: "ec3ecde72560a9fed3b1b83901bdb06d"
server_key_iv: "0eb17d062ba5c60735ef20fc2043aeb0"
server_key_version: "encryptionkey@0"
I20260812 11:23:29.842703 13652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:29.870069 13652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:29.874890 13652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:29.875211 13652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:29.896970 13652 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37699
I20260812 11:23:29.897060 13760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37699 every 8 connection(s)
I20260812 11:23:29.898577 13652 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:23:29.899582 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13652
I20260812 11:23:29.900048 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:23:29.901149 12731 external_mini_cluster.cc:1428] Setting key c614e7cd0f4a83d4f99b92132b979a47
I20260812 11:23:29.904676 13761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 11:23:29 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:29.916658 13761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7: Bootstrap starting.
I20260812 11:23:29.920796 13761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:29.923028 13761 log.cc:824] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:29.927202 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:29.903730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33170 (local address 127.12.110.254:37699)
0812 11:23:29.904686 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:29.904700 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:29.904750 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:23:29.906911 (+ 2161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:29.906935 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:29.907346 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:29.907751 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:29.909359 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:29.910839 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:29.912042 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:29.912705 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:29.912901 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:29.917258 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:29.917265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:29.917279 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:29.917318 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:29.920968 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:29.922284 (+ 1316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:29.922295 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:29.922304 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:29.922693 (+ 389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:29.923336 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:29.923341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:29.923344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:29.924387 (+ 1043us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:29.924786 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:29.925572 (+ 786us) server_negotiation.cc:299] Negotiation successful
0812 11:23:29.926033 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":440,"threads_started":1}
I20260812 11:23:29.927709 13761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7: No bootstrap required, opened a new log
I20260812 11:23:29.936305 13761 raft_consensus.cc:348] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef94a8c4b734efea35efd713eabbba7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37699 } }
I20260812 11:23:29.936707 13761 raft_consensus.cc:374] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:29.936815 13761 raft_consensus.cc:729] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eef94a8c4b734efea35efd713eabbba7, State: Initialized, Role: FOLLOWER
I20260812 11:23:29.937399 13761 consensus_queue.cc:260] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef94a8c4b734efea35efd713eabbba7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37699 } }
I20260812 11:23:29.937634 13761 raft_consensus.cc:388] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:23:29.937796 13761 raft_consensus.cc:482] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:23:29.938084 13761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:29.940452 13761 raft_consensus.cc:504] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef94a8c4b734efea35efd713eabbba7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37699 } }
I20260812 11:23:29.940982 13761 leader_election.cc:302] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eef94a8c4b734efea35efd713eabbba7; no voters:
I20260812 11:23:29.941573 13761 leader_election.cc:288] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:23:29.941769 13768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:29.943068 13768 raft_consensus.cc:686] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [term 1 LEADER]: Becoming Leader. State: Replica: eef94a8c4b734efea35efd713eabbba7, State: Running, Role: LEADER
I20260812 11:23:29.943650 13768 consensus_queue.cc:237] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef94a8c4b734efea35efd713eabbba7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37699 } }
I20260812 11:23:29.944451 13761 sys_catalog.cc:564] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:23:29.946899 13770 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eef94a8c4b734efea35efd713eabbba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef94a8c4b734efea35efd713eabbba7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37699 } } }
I20260812 11:23:29.948241 13770 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:29.947770 13771 sys_catalog.cc:455] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eef94a8c4b734efea35efd713eabbba7. Latest consensus state: current_term: 1 leader_uuid: "eef94a8c4b734efea35efd713eabbba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eef94a8c4b734efea35efd713eabbba7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37699 } } }
I20260812 11:23:29.949299 13771 sys_catalog.cc:458] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:29.950018 13783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:23:29.955394 13783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:23:29.964797 13783 catalog_manager.cc:1347] Generated new cluster ID: 70a6f8d28210493eb7ba67cb398b6620
I20260812 11:23:29.964919 13783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:23:29.980203 13783 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:23:29.981500 13783 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:23:29.998103 13783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eef94a8c4b734efea35efd713eabbba7: Generated new TSK 0
I20260812 11:23:29.999336 13783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:23:30.066358 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37699
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:30.451807 13813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:30.452103 13813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:30.452239 13813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:30.459975 13813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:30.460107 13813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:30.460289 13813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:23:30.470139 13813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37699
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:30.471529 13813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:30.472988 13813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:30.486395 13823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:30.486321 13828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:30.486357 13824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:30.487723 13813 server_base.cc:1034] running on GCE node
I20260812 11:23:30.488927 13813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:30.490244 13813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:30.491778 13813 hybrid_clock.cc:648] HybridClock initialized: now 1786533810491716 us; error 92 us; skew 500 ppm
I20260812 11:23:30.492290 13813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:30 dist-test-slave-h6n0 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533810, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:30.496892 13813 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:23:30.498904 13813 webserver.cc:460] Webserver started at http://127.12.110.193:38489/ using document root <none> and password file <none>
I20260812 11:23:30.499648 13813 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:30.499812 13813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:30.500152 13813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:30.504267 13813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "77e9e5d1078143f79896d9c5ba0bcab2"
format_stamp: "Formatted at 2026-08-12 11:23:30 on dist-test-slave-h6n0"
server_key: "f4d9f199cca62729a291faf45d8536ff"
server_key_iv: "720b6a975563f60160a476b242255dcf"
server_key_version: "encryptionkey@0"
I20260812 11:23:30.505630 13813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "77e9e5d1078143f79896d9c5ba0bcab2"
format_stamp: "Formatted at 2026-08-12 11:23:30 on dist-test-slave-h6n0"
server_key: "f4d9f199cca62729a291faf45d8536ff"
server_key_iv: "720b6a975563f60160a476b242255dcf"
server_key_version: "encryptionkey@0"
I20260812 11:23:30.514037 13813 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.005s
I20260812 11:23:30.518788 13842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:30.520920 13813 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:23:30.521242 13813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "77e9e5d1078143f79896d9c5ba0bcab2"
format_stamp: "Formatted at 2026-08-12 11:23:30 on dist-test-slave-h6n0"
server_key: "f4d9f199cca62729a291faf45d8536ff"
server_key_iv: "720b6a975563f60160a476b242255dcf"
server_key_version: "encryptionkey@0"
I20260812 11:23:30.521553 13813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:30.541903 13813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:30.548029 13813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:30.548363 13813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:30.549438 13813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:30.551110 13813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:30.551220 13813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:30.551383 13813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:30.551491 13813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:30.595547 13813 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:46859
I20260812 11:23:30.595583 14035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:46859 every 8 connection(s)
I20260812 11:23:30.596855 13813 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:23:30.606599 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13813
I20260812 11:23:30.606854 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:23:30.607681 12731 external_mini_cluster.cc:1428] Setting key def3dbb3e68c0d0388bbd0de77af1cd5
Aug 12 11:23:30 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533810, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:30.621208 14039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:30.599665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:34263 (local address 127.12.110.254:37699)
0812 11:23:30.600421 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:30.600427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:30.601233 (+ 806us) server_negotiation.cc:407] Connection header received
0812 11:23:30.603224 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:30.603229 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:30.603317 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:30.603437 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:30.604889 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:30.605656 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:30.608520 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:30.609186 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:30.609337 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:30.613093 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:30.613102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:30.613109 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:30.613157 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:30.617102 (+ 3945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:30.617966 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:30.617974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:30.617979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:30.618149 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:30.619269 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:30.619274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:30.619276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:30.619585 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:30.619763 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:30.620570 (+ 807us) server_negotiation.cc:299] Negotiation successful
0812 11:23:30.620811 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":323,"threads_started":1}
I20260812 11:23:30.622325 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:30.600183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37699 (local address 127.12.110.193:34263)
0812 11:23:30.601025 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:30.601074 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:30.602907 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:30.603647 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:30.603654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:30.604165 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:30.604708 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:30.604726 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:30.605999 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:30.606010 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:30.608324 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:30.608347 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:30.609427 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:30.609433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:30.609641 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:30.610451 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:30.610474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:30.612835 (+ 2361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:30.617292 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:30.617300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:30.617315 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:30.617811 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:30.618417 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:30.618424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:30.618428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:30.618924 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:30.619748 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:30.619757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:30.620151 (+ 394us) client_negotiation.cc:769] Sending connection context
0812 11:23:30.620459 (+ 308us) client_negotiation.cc:240] Negotiation successful
0812 11:23:30.621047 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":388,"threads_started":1}
I20260812 11:23:30.624063 14036 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37699
I20260812 11:23:30.624365 14036 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:30.625110 14036 heartbeater.cc:511] Master 127.12.110.254:37699 requested a full tablet report, sending...
I20260812 11:23:30.627533 13711 ts_manager.cc:194] Registered new tserver with Master: 77e9e5d1078143f79896d9c5ba0bcab2 (127.12.110.193:46859)
I20260812 11:23:30.629081 13711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:34263
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:23:30.666761 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37699
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:31.038331 14049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:31.038707 14049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:31.038817 14049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:31.046602 14049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:31.046814 14049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:31.047001 14049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:23:31.058720 14049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37699
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:31.060016 14049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:31.061331 14049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:31.078055 14061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:31.078080 14062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:31.078132 14064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:31.079087 14049 server_base.cc:1034] running on GCE node
I20260812 11:23:31.079795 14049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:31.080933 14049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:31.082422 14049 hybrid_clock.cc:648] HybridClock initialized: now 1786533811082336 us; error 90 us; skew 500 ppm
I20260812 11:23:31.082806 14049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:31 dist-test-slave-h6n0 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:31.086757 14049 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:23:31.088876 14049 webserver.cc:460] Webserver started at http://127.12.110.194:36723/ using document root <none> and password file <none>
I20260812 11:23:31.089883 14049 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:31.090085 14049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:31.090613 14049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:31.094631 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "02632394e1904d82b6c2db67de2887e7"
format_stamp: "Formatted at 2026-08-12 11:23:31 on dist-test-slave-h6n0"
server_key: "c21222f4d05c8bdd387a8e625391acf1"
server_key_iv: "c7e9328caf3efea5a8563234b1d416bc"
server_key_version: "encryptionkey@0"
I20260812 11:23:31.095568 14049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "02632394e1904d82b6c2db67de2887e7"
format_stamp: "Formatted at 2026-08-12 11:23:31 on dist-test-slave-h6n0"
server_key: "c21222f4d05c8bdd387a8e625391acf1"
server_key_iv: "c7e9328caf3efea5a8563234b1d416bc"
server_key_version: "encryptionkey@0"
I20260812 11:23:31.102023 14049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 11:23:31.106490 14078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:31.108212 14049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:23:31.108604 14049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "02632394e1904d82b6c2db67de2887e7"
format_stamp: "Formatted at 2026-08-12 11:23:31 on dist-test-slave-h6n0"
server_key: "c21222f4d05c8bdd387a8e625391acf1"
server_key_iv: "c7e9328caf3efea5a8563234b1d416bc"
server_key_version: "encryptionkey@0"
I20260812 11:23:31.108927 14049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:31.144677 14049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:31.150651 14049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:31.151417 14049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:31.152575 14049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:31.154217 14049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:31.154292 14049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:31.154457 14049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:31.154565 14049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:31.195154 14049 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:42163
I20260812 11:23:31.195251 14270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:42163 every 8 connection(s)
I20260812 11:23:31.196480 14049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:23:31.205284 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14049
I20260812 11:23:31.205571 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:23:31.206123 12731 external_mini_cluster.cc:1428] Setting key e83808defa76a1f71250a44879bb86db
Aug 12 11:23:31 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:31.218978 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.199348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:35637 (local address 127.12.110.254:37699)
0812 11:23:31.199865 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:31.199869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:31.200802 (+ 933us) server_negotiation.cc:407] Connection header received
0812 11:23:31.202648 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:31.202653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:31.202721 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:31.202838 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:31.205042 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.205884 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.207753 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.208226 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.208383 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.212578 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:31.212583 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:31.212587 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:31.212620 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:31.215536 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:31.216278 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:31.216283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:31.216287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:31.216423 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:31.216854 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:31.216860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:31.216864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:31.217146 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:31.217305 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:31.218117 (+ 812us) server_negotiation.cc:299] Negotiation successful
0812 11:23:31.218502 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
I20260812 11:23:31.220005 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.199816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37699 (local address 127.12.110.194:35637)
0812 11:23:31.200613 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:31.200657 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:31.202335 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:31.203410 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:31.203423 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:31.204253 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:31.204837 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.204856 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.206056 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.206060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.207607 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.207619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.208538 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.208546 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.208833 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.209651 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:31.209691 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:31.212329 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:31.215681 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:31.215691 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:31.215706 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:31.216145 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:31.216542 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:31.216547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:31.216551 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:31.216743 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:31.217278 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:31.217285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:31.217626 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 11:23:31.217897 (+ 271us) client_negotiation.cc:240] Negotiation successful
0812 11:23:31.218706 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":360,"threads_started":1}
I20260812 11:23:31.223166 14271 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37699
I20260812 11:23:31.223694 14271 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:31.224754 14271 heartbeater.cc:511] Master 127.12.110.254:37699 requested a full tablet report, sending...
I20260812 11:23:31.227185 13711 ts_manager.cc:194] Registered new tserver with Master: 02632394e1904d82b6c2db67de2887e7 (127.12.110.194:42163)
I20260812 11:23:31.228469 13711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:35637
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:23:31.270376 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37699
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:23:31.632324 14036 heartbeater.cc:503] Master 127.12.110.254:37699 was elected leader, sending a full tablet report...
W20260812 11:23:31.633616 14287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:31.634179 14287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:31.634584 14287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:31.642181 14287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:31.642287 14287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:31.642403 14287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:23:31.654228 14287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37699
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:31.656036 14287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:31.657874 14287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:31.675495 14308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:31.675473 14311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:31.675522 14307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:31.676272 14287 server_base.cc:1034] running on GCE node
I20260812 11:23:31.676939 14287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:31.677946 14287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:31.679196 14287 hybrid_clock.cc:648] HybridClock initialized: now 1786533811679155 us; error 30 us; skew 500 ppm
I20260812 11:23:31.679692 14287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:31 dist-test-slave-h6n0 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:31.683314 14287 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:23:31.684988 14287 webserver.cc:460] Webserver started at http://127.12.110.195:44263/ using document root <none> and password file <none>
I20260812 11:23:31.685680 14287 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:31.685786 14287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:31.686069 14287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:31.690310 14287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "b24c0089ae124d73ab2cac69cb0b58ba"
format_stamp: "Formatted at 2026-08-12 11:23:31 on dist-test-slave-h6n0"
server_key: "e47563f8bd5c3188827c46dfaab08e86"
server_key_iv: "0c3e315e11df0c0ce3af328edbac3c64"
server_key_version: "encryptionkey@0"
I20260812 11:23:31.691527 14287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "b24c0089ae124d73ab2cac69cb0b58ba"
format_stamp: "Formatted at 2026-08-12 11:23:31 on dist-test-slave-h6n0"
server_key: "e47563f8bd5c3188827c46dfaab08e86"
server_key_iv: "0c3e315e11df0c0ce3af328edbac3c64"
server_key_version: "encryptionkey@0"
I20260812 11:23:31.698515 14287 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260812 11:23:31.702414 14322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:31.704478 14287 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:23:31.704713 14287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "b24c0089ae124d73ab2cac69cb0b58ba"
format_stamp: "Formatted at 2026-08-12 11:23:31 on dist-test-slave-h6n0"
server_key: "e47563f8bd5c3188827c46dfaab08e86"
server_key_iv: "0c3e315e11df0c0ce3af328edbac3c64"
server_key_version: "encryptionkey@0"
I20260812 11:23:31.704979 14287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:31.739328 14287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:31.745072 14287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:31.745451 14287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:31.747087 14287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:31.748653 14287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:31.748734 14287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:31.748988 14287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:31.749073 14287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:31.788439 14287 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:42237
I20260812 11:23:31.788484 14505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:42237 every 8 connection(s)
I20260812 11:23:31.789784 14287 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:23:31.795957 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14287
I20260812 11:23:31.796175 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:23:31.796754 12731 external_mini_cluster.cc:1428] Setting key ce5f49d297761ba2a8566cf5809aa4ac
Aug 12 11:23:31 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533811, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:31.812745 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.792503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:56901 (local address 127.12.110.254:37699)
0812 11:23:31.793031 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:31.793036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:31.793709 (+ 673us) server_negotiation.cc:407] Connection header received
0812 11:23:31.795705 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:31.795710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:31.795793 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:31.795892 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:31.797842 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.798878 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.800690 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.801211 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.801336 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.805606 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:31.805612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:31.805615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:31.805662 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:31.808819 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:31.809681 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:31.809687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:31.809690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:31.809819 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:31.810414 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:31.810421 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:31.810424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:31.811000 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:31.811322 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:31.812134 (+ 812us) server_negotiation.cc:299] Negotiation successful
0812 11:23:31.812381 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":282,"threads_started":1}
I20260812 11:23:31.814391 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.792862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37699 (local address 127.12.110.195:56901)
0812 11:23:31.793549 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:31.793596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:31.795390 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:31.796183 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:31.796197 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:31.796939 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:31.797628 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.797644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.799163 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.799169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.800522 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.800540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.801429 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.801434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.801636 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.802402 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:31.802424 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:31.805269 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:31.809012 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:31.809021 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:31.809038 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:31.809537 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:31.809941 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:31.809946 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:31.809949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:31.810246 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:31.811243 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:31.811251 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:31.811630 (+ 379us) client_negotiation.cc:769] Sending connection context
0812 11:23:31.811957 (+ 327us) client_negotiation.cc:240] Negotiation successful
0812 11:23:31.812480 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":295,"threads_started":1}
I20260812 11:23:31.815840 14506 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37699
I20260812 11:23:31.816236 14506 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:31.816988 14506 heartbeater.cc:511] Master 127.12.110.254:37699 requested a full tablet report, sending...
I20260812 11:23:31.818915 13711 ts_manager.cc:194] Registered new tserver with Master: b24c0089ae124d73ab2cac69cb0b58ba (127.12.110.195:42237)
I20260812 11:23:31.819774 13711 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:56901
I20260812 11:23:31.827507 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:23:31.849649 14508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.834476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.12.110.254:37699)
0812 11:23:31.834720 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:31.834729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:31.835609 (+ 880us) server_negotiation.cc:407] Connection header received
0812 11:23:31.835730 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:31.835734 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:31.835812 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:31.835900 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:31.837631 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.838395 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.839822 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.840324 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.840479 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.843634 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:31.843643 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:31.843648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:31.843692 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:31.846923 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:31.847673 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:31.847677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:31.847679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:31.847792 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:31.848206 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:31.848210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:31.848212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:31.848560 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:31.848765 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:31.849114 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 11:23:31.849286 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260812 11:23:31.856910 13711 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:23:31.864212 13711 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:31.903359 14538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.881161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.12.110.195:42237)
0812 11:23:31.894493 (+ 13332us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:31.894505 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:31.894570 (+ 65us) server_negotiation.cc:407] Connection header received
0812 11:23:31.894692 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:31.894702 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:31.894972 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:31.895210 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:31.895821 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.897520 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.900484 (+ 2964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.901868 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.902036 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.902196 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:31.902792 (+ 596us) server_negotiation.cc:299] Negotiation successful
0812 11:23:31.902958 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13136,"thread_start_us":12951,"threads_started":1}
I20260812 11:23:31.904138 14529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.880699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:42237 (local address 127.0.0.1:50444)
0812 11:23:31.893443 (+ 12744us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:31.893467 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:31.893607 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:31.895211 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:31.895216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:31.895243 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:31.895615 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.895625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.897709 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.897715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.900281 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.900308 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.902391 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.902396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.902593 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.902623 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:23:31.903560 (+ 937us) client_negotiation.cc:240] Negotiation successful
0812 11:23:31.903691 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12577,"thread_start_us":8649,"threads_started":1}
I20260812 11:23:31.904736 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.878423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52332 (local address 127.12.110.194:42163)
0812 11:23:31.892462 (+ 14039us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:31.892471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:31.892498 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:23:31.892604 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:31.892610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:31.892790 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:31.893247 (+ 457us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:31.894130 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.895624 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.899965 (+ 4341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.901622 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.901784 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.901976 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:31.903819 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0812 11:23:31.903977 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13862,"thread_start_us":13928,"threads_started":1}
I20260812 11:23:31.905390 14537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.878287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:42163 (local address 127.0.0.1:52332)
0812 11:23:31.889777 (+ 11490us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:31.889816 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:31.890017 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:31.893248 (+ 3231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:31.893261 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:31.893299 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:31.893720 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.893741 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.895796 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.895800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.899739 (+ 3939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.899756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.902126 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.902132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.902388 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.902440 (+ 52us) client_negotiation.cc:769] Sending connection context
0812 11:23:31.902603 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 11:23:31.905034 (+ 2431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11240,"thread_start_us":15425,"threads_started":1}
I20260812 11:23:31.905948 14530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.878332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46859 (local address 127.0.0.1:44286)
0812 11:23:31.892145 (+ 13813us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:31.892170 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:31.892285 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:31.896742 (+ 4457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:31.896746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:31.896782 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:31.897458 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.897476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.899556 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.899563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.901916 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:31.901932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.904878 (+ 2946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.904886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:31.905064 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.905101 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:23:31.905215 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 11:23:31.905350 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13675,"thread_start_us":13502,"threads_started":1}
I20260812 11:23:31.906209 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:31.878781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44286 (local address 127.12.110.193:46859)
0812 11:23:31.896022 (+ 17241us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:31.896028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:31.896049 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:23:31.896119 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:31.896124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:31.896306 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:31.896491 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:31.897705 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.899351 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.902117 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:31.904232 (+ 2115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:31.904530 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:31.904713 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:31.905294 (+ 581us) server_negotiation.cc:299] Negotiation successful
0812 11:23:31.905692 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17076,"thread_start_us":16894,"threads_started":1}
I20260812 11:23:31.908934 14160 tablet_service.cc:1467] Processing CreateTablet for tablet db396489b9454399b76b7183bf93136c (DEFAULT_TABLE table=test-table [id=3936d0e19e494ba2ad74cf552f41f8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:31.910687 14160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db396489b9454399b76b7183bf93136c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:31.912616 14403 tablet_service.cc:1467] Processing CreateTablet for tablet db396489b9454399b76b7183bf93136c (DEFAULT_TABLE table=test-table [id=3936d0e19e494ba2ad74cf552f41f8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:31.913167 13932 tablet_service.cc:1467] Processing CreateTablet for tablet db396489b9454399b76b7183bf93136c (DEFAULT_TABLE table=test-table [id=3936d0e19e494ba2ad74cf552f41f8a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:31.914136 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db396489b9454399b76b7183bf93136c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:31.914548 13932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db396489b9454399b76b7183bf93136c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:31.926142 14539 tablet_bootstrap.cc:492] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7: Bootstrap starting.
I20260812 11:23:31.930505 14539 tablet_bootstrap.cc:654] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:31.931113 14541 tablet_bootstrap.cc:492] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2: Bootstrap starting.
I20260812 11:23:31.932183 14539 log.cc:824] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:31.932686 14540 tablet_bootstrap.cc:492] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba: Bootstrap starting.
I20260812 11:23:31.934777 14541 tablet_bootstrap.cc:654] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:31.935930 14540 tablet_bootstrap.cc:654] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:31.936007 14541 log.cc:824] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:31.936277 14539 tablet_bootstrap.cc:492] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7: No bootstrap required, opened a new log
I20260812 11:23:31.936589 14539 ts_tablet_manager.cc:1397] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260812 11:23:31.937338 14540 log.cc:824] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:31.939039 14541 tablet_bootstrap.cc:492] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2: No bootstrap required, opened a new log
I20260812 11:23:31.939301 14541 ts_tablet_manager.cc:1397] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260812 11:23:31.940340 14540 tablet_bootstrap.cc:492] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba: No bootstrap required, opened a new log
I20260812 11:23:31.940589 14540 ts_tablet_manager.cc:1397] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260812 11:23:31.945772 14539 raft_consensus.cc:348] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:31.946445 14539 raft_consensus.cc:374] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:31.946621 14539 raft_consensus.cc:729] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02632394e1904d82b6c2db67de2887e7, State: Initialized, Role: FOLLOWER
I20260812 11:23:31.947535 14539 consensus_queue.cc:260] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:31.949940 14539 ts_tablet_manager.cc:1428] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260812 11:23:31.950098 14271 heartbeater.cc:503] Master 127.12.110.254:37699 was elected leader, sending a full tablet report...
I20260812 11:23:31.950671 14541 raft_consensus.cc:348] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:31.951110 14541 raft_consensus.cc:374] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:31.951215 14541 raft_consensus.cc:729] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e9e5d1078143f79896d9c5ba0bcab2, State: Initialized, Role: FOLLOWER
I20260812 11:23:31.951674 14540 raft_consensus.cc:348] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:31.952085 14540 raft_consensus.cc:374] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:31.951930 14541 consensus_queue.cc:260] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:31.952193 14540 raft_consensus.cc:729] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24c0089ae124d73ab2cac69cb0b58ba, State: Initialized, Role: FOLLOWER
I20260812 11:23:31.952742 14540 consensus_queue.cc:260] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:31.953811 14541 ts_tablet_manager.cc:1428] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260812 11:23:31.954787 14506 heartbeater.cc:503] Master 127.12.110.254:37699 was elected leader, sending a full tablet report...
I20260812 11:23:31.955346 14540 ts_tablet_manager.cc:1428] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260812 11:23:32.032783 14549 raft_consensus.cc:482] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:32.033417 14549 raft_consensus.cc:504] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:32.035889 14549 leader_election.cc:288] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02632394e1904d82b6c2db67de2887e7 (127.12.110.194:42163), 77e9e5d1078143f79896d9c5ba0bcab2 (127.12.110.193:46859)
W20260812 11:23:32.041711 14507 tablet.cc:2369] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:23:32.043799 14511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.036424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:42163 (local address 127.12.110.195:54659)
0812 11:23:32.036736 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:32.036752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:32.036916 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:32.037237 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:32.037242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:32.037267 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:32.037500 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:32.037512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.039069 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.039075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:32.041441 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:32.041460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.043142 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.043148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:32.043328 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.043354 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:23:32.043470 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:23:32.043593 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260812 11:23:32.044035 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.036398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:54659 (local address 127.12.110.194:42163)
0812 11:23:32.036641 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.036649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.036838 (+ 189us) server_negotiation.cc:407] Connection header received
0812 11:23:32.037045 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.037051 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.037118 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.037220 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:32.037643 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.038898 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.041674 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.042946 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.043123 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.043215 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.043564 (+ 349us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.043739 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:23:32.045421 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.036398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:40997 (local address 127.12.110.193:46859)
0812 11:23:32.036614 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.036622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.037413 (+ 791us) server_negotiation.cc:407] Connection header received
0812 11:23:32.037534 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.037541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.037627 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.037707 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:32.038619 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.039768 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.042873 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.044254 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.044565 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.044655 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.045088 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.045186 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:23:32.045663 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.036492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46859 (local address 127.12.110.195:40997)
0812 11:23:32.037309 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:32.037326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:32.037433 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:32.037759 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:32.037764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:32.037792 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:32.038290 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:32.038304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.040162 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.040170 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:32.042577 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:32.042594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.044622 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.044627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:32.044878 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.044921 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 11:23:32.045053 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 11:23:32.045233 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":481,"threads_started":1}
I20260812 11:23:32.045543 14200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db396489b9454399b76b7183bf93136c" candidate_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02632394e1904d82b6c2db67de2887e7" is_pre_election: true
I20260812 11:23:32.046412 14200 raft_consensus.cc:2454] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24c0089ae124d73ab2cac69cb0b58ba in term 0.
I20260812 11:23:32.046908 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db396489b9454399b76b7183bf93136c" candidate_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" is_pre_election: true
I20260812 11:23:32.047473 14323 leader_election.cc:302] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02632394e1904d82b6c2db67de2887e7, b24c0089ae124d73ab2cac69cb0b58ba; no voters:
I20260812 11:23:32.047561 13963 raft_consensus.cc:2454] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b24c0089ae124d73ab2cac69cb0b58ba in term 0.
I20260812 11:23:32.047907 14549 raft_consensus.cc:2781] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:32.048090 14549 raft_consensus.cc:482] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:32.048228 14549 raft_consensus.cc:3037] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:32.050628 14549 raft_consensus.cc:504] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:32.051560 14549 leader_election.cc:288] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [CANDIDATE]: Term 1 election: Requested vote from peers 02632394e1904d82b6c2db67de2887e7 (127.12.110.194:42163), 77e9e5d1078143f79896d9c5ba0bcab2 (127.12.110.193:46859)
I20260812 11:23:32.052065 13963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db396489b9454399b76b7183bf93136c" candidate_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77e9e5d1078143f79896d9c5ba0bcab2"
I20260812 11:23:32.052093 14200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db396489b9454399b76b7183bf93136c" candidate_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02632394e1904d82b6c2db67de2887e7"
I20260812 11:23:32.052277 13963 raft_consensus.cc:3037] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:32.052330 14200 raft_consensus.cc:3037] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:32.054628 14200 raft_consensus.cc:2454] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24c0089ae124d73ab2cac69cb0b58ba in term 1.
I20260812 11:23:32.055085 14323 leader_election.cc:302] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02632394e1904d82b6c2db67de2887e7, b24c0089ae124d73ab2cac69cb0b58ba; no voters:
I20260812 11:23:32.055279 13963 raft_consensus.cc:2454] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b24c0089ae124d73ab2cac69cb0b58ba in term 1.
I20260812 11:23:32.055474 14549 raft_consensus.cc:2781] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:32.056175 14549 raft_consensus.cc:686] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [term 1 LEADER]: Becoming Leader. State: Replica: b24c0089ae124d73ab2cac69cb0b58ba, State: Running, Role: LEADER
I20260812 11:23:32.056782 14549 consensus_queue.cc:237] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } }
I20260812 11:23:32.061935 13711 catalog_manager.cc:5495] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba reported cstate change: term changed from 0 to 1, leader changed from <none> to b24c0089ae124d73ab2cac69cb0b58ba (127.12.110.195). New cstate: current_term: 1 leader_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02632394e1904d82b6c2db67de2887e7" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77e9e5d1078143f79896d9c5ba0bcab2" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b24c0089ae124d73ab2cac69cb0b58ba" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42237 } health_report { overall_health: HEALTHY } } }
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:32.098598 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.081736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.12.110.193:46859)
0812 11:23:32.082054 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.082063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.082626 (+ 563us) server_negotiation.cc:407] Connection header received
0812 11:23:32.082860 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.082865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.082931 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.083049 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:32.084767 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.085559 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.086888 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.087298 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.087463 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.091699 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:32.091712 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:32.091727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:32.091762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:32.094640 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.095455 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.095467 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.095470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.095793 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.096600 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.096605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.096607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.097168 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:32.097580 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.097899 (+ 319us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.098133 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
W20260812 11:23:32.099947 14038 tablet.cc:2369] T db396489b9454399b76b7183bf93136c P 77e9e5d1078143f79896d9c5ba0bcab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:32.157158 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.141350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44312 (local address 127.12.110.193:46859)
0812 11:23:32.141605 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.141616 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.141987 (+ 371us) server_negotiation.cc:407] Connection header received
0812 11:23:32.142109 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.142114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.142182 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.142270 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:32.143967 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.144923 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.146519 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.147101 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.147208 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.151487 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:32.151493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:32.151496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:32.151526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:32.154266 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.155022 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.155028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.155031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.155308 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.155721 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.155725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.155727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.156029 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:32.156249 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.156553 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.156791 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[13630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260812 11:23:32.198819 14277 tablet.cc:2369] T db396489b9454399b76b7183bf93136c P 02632394e1904d82b6c2db67de2887e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:32.223532 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.202907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.12.110.193:46859)
0812 11:23:32.203121 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.203131 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.203448 (+ 317us) server_negotiation.cc:407] Connection header received
0812 11:23:32.203656 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.203665 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.203730 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.203852 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:32.205178 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.205890 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.207209 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.207639 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.207774 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.217576 (+ 9802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:32.217581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:32.217585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:32.217616 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:32.220352 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.221444 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.221449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.221451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.221589 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.222055 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.222059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.222061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.222405 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:32.222722 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.223039 (+ 317us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.223230 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260812 11:23:32.226033 13902 tablet_service.cc:2908] Rejecting scan request for tablet db396489b9454399b76b7183bf93136c: Uninitialized: clean time has not yet been initialized
I20260812 11:23:32.230227 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13813
I20260812 11:23:32.249607 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.231176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.12.110.193:46859)
0812 11:23:32.231407 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.231413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.231665 (+ 252us) server_negotiation.cc:407] Connection header received
0812 11:23:32.231770 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.231773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.231841 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.231921 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:32.233570 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.234692 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.236978 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.237745 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.237910 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.241176 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:32.241183 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:32.241188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:32.241231 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:32.246383 (+ 5152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.247291 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.247300 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.247304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.247458 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.247876 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.247882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.247885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.248206 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:32.248523 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.248834 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.249073 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:23:32.250720 14031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:32.358994 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13813
I20260812 11:23:32.391074 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14049
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:23:32.407651 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.392149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52342 (local address 127.12.110.194:42163)
0812 11:23:32.392462 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.392468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.392487 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:23:32.392545 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.392549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.392628 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.392718 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:32.393988 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.394912 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.396247 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.396647 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.396749 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.401144 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:32.401159 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:32.401175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:32.401209 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:32.404370 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.405294 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.405302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.405305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.405489 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.405952 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.405959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.405962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.406387 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:32.406780 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.407053 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.407289 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260812 11:23:32.408424 14265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:32.462754 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.461619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46859 (local address 127.12.110.195:51609)
0812 11:23:32.461824 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:32.461952 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260812 11:23:32.463346 14325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46859: connect: Connection refused (error 111)
I20260812 11:23:32.465272 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.464728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46859 (local address 127.12.110.195:40863)
0812 11:23:32.464936 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:32.465035 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46859: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260812 11:23:32.465560 14325 consensus_peers.cc:487] T db396489b9454399b76b7183bf93136c P b24c0089ae124d73ab2cac69cb0b58ba -> Peer 77e9e5d1078143f79896d9c5ba0bcab2 (127.12.110.193:46859): Couldn't send request to peer 77e9e5d1078143f79896d9c5ba0bcab2. Status: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 11:23:32.505671 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14049
I20260812 11:23:32.533157 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14287
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[13630](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:23:32.552405 14583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.533911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50452 (local address 127.12.110.195:42237)
0812 11:23:32.534496 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:32.534500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:32.534520 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:23:32.534635 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:32.534639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:32.534707 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:32.534823 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:32.536098 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.536814 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.538115 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:32.538764 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:32.538904 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:32.543814 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:32.543828 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:32.543846 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:32.543877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:32.548467 (+ 4590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.549293 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.549302 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.549305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.549483 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:32.550192 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:32.550197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:32.550199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:32.550761 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:32.551087 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:32.551543 (+ 456us) server_negotiation.cc:299] Negotiation successful
0812 11:23:32.551804 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":303,"threads_started":1}
I20260812 11:23:32.553128 14500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:32.646639 14550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:32.646009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:42163 (local address 127.12.110.195:35589)
0812 11:23:32.646253 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:32.646391 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.194:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 11:23:32.646749 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14287
I20260812 11:23:32.669907 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13652
I20260812 11:23:32.671532 13755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:32.802335 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13652
2026-08-12T11:23:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5057 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[14591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[14591](info): set up 2 sockets
Aug 12 11:23:32 dist-test-slave-h6n0 krb5kdc[14591](info): commencing operation
krb5kdc: starting...
W20260812 11:23:34.243206 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.384s user 0.003s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:23:34 dist-test-slave-h6n0 krb5kdc[14591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:23:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:23:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:23:34.492216 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36401
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:34.856824 14611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:34.857260 14611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:34.857430 14611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:34.864897 14611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:23:34.865051 14611 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:23:34.865154 14611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:34.865278 14611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:34.865366 14611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:23:34.865461 14611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:23:34.875703 14611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36401
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:34.877108 14611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:34.878517 14611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:34.892138 14618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:34.893440 14619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:34.893518 14622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:34.894515 14611 server_base.cc:1034] running on GCE node
I20260812 11:23:34.895666 14611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:34.899050 14611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:34.901032 14611 hybrid_clock.cc:648] HybridClock initialized: now 1786533814900969 us; error 152 us; skew 500 ppm
I20260812 11:23:34.901685 14611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:34 dist-test-slave-h6n0 krb5kdc[14591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533814, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:34.907939 14611 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:23:34.909883 14611 webserver.cc:460] Webserver started at http://127.12.110.254:45423/ using document root <none> and password file <none>
I20260812 11:23:34.910781 14611 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:34.910926 14611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:34.911311 14611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:34.915930 14611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "4f7ea935c8e5425184dce26661ef8728"
format_stamp: "Formatted at 2026-08-12 11:23:34 on dist-test-slave-h6n0"
server_key: "2fa07c117283f638def7b0f4384e13de"
server_key_iv: "0bff5a609d2f5b95a0165baf496c14dc"
server_key_version: "encryptionkey@0"
I20260812 11:23:34.917174 14611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "4f7ea935c8e5425184dce26661ef8728"
format_stamp: "Formatted at 2026-08-12 11:23:34 on dist-test-slave-h6n0"
server_key: "2fa07c117283f638def7b0f4384e13de"
server_key_iv: "0bff5a609d2f5b95a0165baf496c14dc"
server_key_version: "encryptionkey@0"
I20260812 11:23:34.924953 14611 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 11:23:34.929729 14638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:34.931449 14611 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 11:23:34.931668 14611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "4f7ea935c8e5425184dce26661ef8728"
format_stamp: "Formatted at 2026-08-12 11:23:34 on dist-test-slave-h6n0"
server_key: "2fa07c117283f638def7b0f4384e13de"
server_key_iv: "0bff5a609d2f5b95a0165baf496c14dc"
server_key_version: "encryptionkey@0"
I20260812 11:23:34.931932 14611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:34.969964 14611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:34.974804 14611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:34.975111 14611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:35.002840 14611 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:36401
I20260812 11:23:35.002945 14735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:36401 every 8 connection(s)
I20260812 11:23:35.004460 14611 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:23:35.008404 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:35.010737 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14611
I20260812 11:23:35.010983 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:23:35.011821 12731 external_mini_cluster.cc:1428] Setting key 058a563b58a9dc12f4dd9ade126439f4
I20260812 11:23:35.024477 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728: Bootstrap starting.
I20260812 11:23:35.027712 14736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728: Neither blocks nor log segments found. Creating new log.
Aug 12 11:23:35 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:35.028829 14736 log.cc:824] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:35.032070 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728: No bootstrap required, opened a new log
I20260812 11:23:35.038058 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:35.014230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.12.110.254:36401)
0812 11:23:35.015188 (+ 958us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:35.015200 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:35.015242 (+ 42us) server_negotiation.cc:407] Connection header received
0812 11:23:35.016907 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:35.016929 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:35.017311 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:35.017628 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:35.019476 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.021692 (+ 2216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.023762 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.024541 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.024791 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:35.029750 (+ 4959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:35.029759 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:35.029773 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:35.029813 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:35.033556 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:35.034291 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:35.034297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:35.034303 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:35.034510 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:35.034988 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:35.034994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:35.034997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:35.035730 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:35.036074 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:35.036505 (+ 431us) server_negotiation.cc:299] Negotiation successful
0812 11:23:35.036979 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":514,"threads_started":1}
I20260812 11:23:35.041764 14736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ea935c8e5425184dce26661ef8728" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36401 } }
I20260812 11:23:35.042135 14736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:35.042244 14736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f7ea935c8e5425184dce26661ef8728, State: Initialized, Role: FOLLOWER
I20260812 11:23:35.042878 14736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ea935c8e5425184dce26661ef8728" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36401 } }
I20260812 11:23:35.043136 14736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:23:35.043298 14736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:23:35.043497 14736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:35.045778 14736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ea935c8e5425184dce26661ef8728" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36401 } }
I20260812 11:23:35.046298 14736 leader_election.cc:302] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f7ea935c8e5425184dce26661ef8728; no voters:
I20260812 11:23:35.046962 14736 leader_election.cc:288] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:23:35.047135 14743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:35.049002 14743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [term 1 LEADER]: Becoming Leader. State: Replica: 4f7ea935c8e5425184dce26661ef8728, State: Running, Role: LEADER
I20260812 11:23:35.049561 14743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ea935c8e5425184dce26661ef8728" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36401 } }
I20260812 11:23:35.050076 14736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:23:35.052826 14748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f7ea935c8e5425184dce26661ef8728. Latest consensus state: current_term: 1 leader_uuid: "4f7ea935c8e5425184dce26661ef8728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ea935c8e5425184dce26661ef8728" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36401 } } }
I20260812 11:23:35.053356 14748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:35.054768 14746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f7ea935c8e5425184dce26661ef8728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f7ea935c8e5425184dce26661ef8728" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36401 } } }
I20260812 11:23:35.055298 14746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:35.060832 14759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:23:35.067853 14759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:23:35.077124 14759 catalog_manager.cc:1347] Generated new cluster ID: f1c81040322040d1b4e735b23f1861d5
I20260812 11:23:35.077313 14759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:23:35.097347 14759 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:23:35.098923 14759 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:23:35.111119 14759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728: Generated new TSK 0
I20260812 11:23:35.112457 14759 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:23:35.174620 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36401
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:35.561574 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:35.561975 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:35.562093 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:35.571367 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:35.571514 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:35.571611 14785 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:23:35.571702 14785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:23:35.581851 14785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:36401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:35.583217 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:35.584470 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:35.598084 14792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:35.598160 14793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:35.598443 14798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:35.599251 14785 server_base.cc:1034] running on GCE node
I20260812 11:23:35.599896 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:35.600912 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:35.602254 14785 hybrid_clock.cc:648] HybridClock initialized: now 1786533815602146 us; error 55 us; skew 500 ppm
I20260812 11:23:35.602793 14785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:35 dist-test-slave-h6n0 krb5kdc[14591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533815, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:35.606715 14785 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:23:35.608544 14785 webserver.cc:460] Webserver started at http://127.12.110.193:44633/ using document root <none> and password file <none>
I20260812 11:23:35.609521 14785 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:35.609650 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:35.609972 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:35.613580 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "babd9b5f948441d8ba0df35635c86400"
format_stamp: "Formatted at 2026-08-12 11:23:35 on dist-test-slave-h6n0"
server_key: "f652360ad217e99139635a1c0a08f8e3"
server_key_iv: "67f134bb6201179c2ab5342b2d628085"
server_key_version: "encryptionkey@0"
I20260812 11:23:35.614573 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "babd9b5f948441d8ba0df35635c86400"
format_stamp: "Formatted at 2026-08-12 11:23:35 on dist-test-slave-h6n0"
server_key: "f652360ad217e99139635a1c0a08f8e3"
server_key_iv: "67f134bb6201179c2ab5342b2d628085"
server_key_version: "encryptionkey@0"
I20260812 11:23:35.619935 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:23:35.623673 14809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:35.625025 14785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:23:35.625219 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "babd9b5f948441d8ba0df35635c86400"
format_stamp: "Formatted at 2026-08-12 11:23:35 on dist-test-slave-h6n0"
server_key: "f652360ad217e99139635a1c0a08f8e3"
server_key_iv: "67f134bb6201179c2ab5342b2d628085"
server_key_version: "encryptionkey@0"
I20260812 11:23:35.625466 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:35.641767 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:35.647715 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:35.648033 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:35.651930 14785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:35.652050 14785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:35.652237 14785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:35.652348 14785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:23:35 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533815, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:35.683439 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:35.653985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36401 (local address 127.12.110.193:36897)
0812 11:23:35.655561 (+ 1576us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:35.655629 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:35.658503 (+ 2874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:35.659569 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:35.659586 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:35.660513 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:35.661383 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:35.661423 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.663200 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.663208 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:35.666569 (+ 3361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:35.666593 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.667983 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.667988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:35.668202 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:35.669153 (+ 951us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:35.669175 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:35.672157 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:35.676389 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:35.676412 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:35.676450 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:35.677767 (+ 1317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:35.678260 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:35.678266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:35.678276 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:35.678648 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:35.679445 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:35.679452 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:35.680281 (+ 829us) client_negotiation.cc:769] Sending connection context
0812 11:23:35.680660 (+ 379us) client_negotiation.cc:240] Negotiation successful
0812 11:23:35.681365 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1189,"thread_start_us":651,"threads_started":1}
I20260812 11:23:35.683723 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:35.653088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:36897 (local address 127.12.110.254:36401)
0812 11:23:35.653560 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:35.653564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:35.655846 (+ 2282us) server_negotiation.cc:407] Connection header received
0812 11:23:35.658961 (+ 3115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:35.658970 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:35.659100 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:35.659254 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:35.661628 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.662992 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.666764 (+ 3772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.667713 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.667904 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:35.672388 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:35.672393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:35.672396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:35.672428 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:35.676156 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:35.677953 (+ 1797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:35.677960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:35.677965 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:35.678125 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:35.678834 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:35.678838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:35.678840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:35.679209 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:35.679365 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:35.682864 (+ 3499us) server_negotiation.cc:299] Negotiation successful
0812 11:23:35.683172 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":271,"threads_started":1}
I20260812 11:23:35.712517 14785 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:39685
I20260812 11:23:35.712528 15012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:39685 every 8 connection(s)
I20260812 11:23:35.714449 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:23:35.715801 14828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:35.694774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:37365 (local address 127.12.110.254:36401)
0812 11:23:35.695218 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:35.695229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:35.696760 (+ 1531us) server_negotiation.cc:407] Connection header received
0812 11:23:35.696989 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:35.697000 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:35.697117 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:35.697271 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:35.699211 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.700516 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.704119 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.704974 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.705149 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:35.708115 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:35.708120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:35.708123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:35.708157 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:35.711786 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:35.712671 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:35.712677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:35.712681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:35.712889 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:35.713816 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:35.713822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:35.713826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:35.714224 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:35.714473 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:35.715028 (+ 555us) server_negotiation.cc:299] Negotiation successful
0812 11:23:35.715365 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260812 11:23:35.715914 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:35.695505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36401 (local address 127.12.110.193:37365)
0812 11:23:35.696562 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:35.696595 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:35.696818 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:35.697372 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:35.697380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:35.698395 (+ 1015us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:35.698976 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:35.698992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.700947 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.700959 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:35.703830 (+ 2871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:35.703856 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:35.705274 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:35.705290 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:35.705499 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:35.706432 (+ 933us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:35.706466 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:35.707853 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:35.711939 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:35.711950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:35.711956 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:35.712521 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:35.713396 (+ 875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:35.713403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:35.713410 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:35.713614 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:35.714380 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:35.714385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:35.714716 (+ 331us) client_negotiation.cc:769] Sending connection context
0812 11:23:35.714890 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 11:23:35.715387 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":711,"threads_started":1}
I20260812 11:23:35.718458 15013 heartbeater.cc:347] Connected to a master server at 127.12.110.254:36401
I20260812 11:23:35.719030 15013 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:35.720039 15013 heartbeater.cc:511] Master 127.12.110.254:36401 requested a full tablet report, sending...
I20260812 11:23:35.721550 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14785
I20260812 11:23:35.721833 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:23:35.722831 12731 external_mini_cluster.cc:1428] Setting key dc781c20f83dc3bb134970362022d2c9
I20260812 11:23:35.723027 14666 ts_manager.cc:194] Registered new tserver with Master: babd9b5f948441d8ba0df35635c86400 (127.12.110.193:39685)
I20260812 11:23:35.724989 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:36897
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:23:35.782760 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36401
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:36.144997 15018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:36.145380 15018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:36.145573 15018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:36.155625 15018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:36.155831 15018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:36.155989 15018 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:23:36.156133 15018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:23:36.167001 15018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:36401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:36.169001 15018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:36.170998 15018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:36.185510 15026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:36.185920 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:36.185884 15031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:36.187026 15018 server_base.cc:1034] running on GCE node
I20260812 11:23:36.187649 15018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:36.188696 15018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:36.190135 15018 hybrid_clock.cc:648] HybridClock initialized: now 1786533816189999 us; error 110 us; skew 500 ppm
I20260812 11:23:36.190461 15018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:36 dist-test-slave-h6n0 krb5kdc[14591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:36.193943 15018 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:23:36.195441 15018 webserver.cc:460] Webserver started at http://127.12.110.194:39313/ using document root <none> and password file <none>
I20260812 11:23:36.196120 15018 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:36.196221 15018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:36.196489 15018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:36.199532 15018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "b403c8f308b047f48b505f98ce992a46"
format_stamp: "Formatted at 2026-08-12 11:23:36 on dist-test-slave-h6n0"
server_key: "8d3bff904edc486710884ef72afaf339"
server_key_iv: "85ba5042395d59e7fada9f2aeab79f61"
server_key_version: "encryptionkey@0"
I20260812 11:23:36.200457 15018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "b403c8f308b047f48b505f98ce992a46"
format_stamp: "Formatted at 2026-08-12 11:23:36 on dist-test-slave-h6n0"
server_key: "8d3bff904edc486710884ef72afaf339"
server_key_iv: "85ba5042395d59e7fada9f2aeab79f61"
server_key_version: "encryptionkey@0"
I20260812 11:23:36.205981 15018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 11:23:36.210088 15043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:36.212203 15018 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:23:36.212375 15018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "b403c8f308b047f48b505f98ce992a46"
format_stamp: "Formatted at 2026-08-12 11:23:36 on dist-test-slave-h6n0"
server_key: "8d3bff904edc486710884ef72afaf339"
server_key_iv: "85ba5042395d59e7fada9f2aeab79f61"
server_key_version: "encryptionkey@0"
I20260812 11:23:36.212580 15018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:36.232798 15018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:36.237929 15018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:36.238308 15018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:36.241662 15018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:36.241752 15018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:36.241971 15018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:36.242084 15018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:23:36 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:36.268321 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.242176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:52777 (local address 127.12.110.254:36401)
0812 11:23:36.242681 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:36.242688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:36.243895 (+ 1207us) server_negotiation.cc:407] Connection header received
0812 11:23:36.246130 (+ 2235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:36.246139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:36.246255 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:36.246393 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:36.248601 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.250014 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.252871 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.253301 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.253443 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.258694 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:36.258703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:36.258710 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:36.258753 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:36.262160 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.263133 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.263141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.263145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.263288 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.263973 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.263981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.263987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.264429 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:36.264629 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:36.267451 (+ 2822us) server_negotiation.cc:299] Negotiation successful
0812 11:23:36.267781 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260812 11:23:36.269237 15059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.242822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36401 (local address 127.12.110.194:52777)
0812 11:23:36.243713 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:36.243757 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:36.245795 (+ 2038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:36.246702 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:36.246713 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:36.247465 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:36.248369 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.248404 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.250203 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.250211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.252682 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.252697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.253880 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.253893 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.254189 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.255061 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:36.255112 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:36.258317 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:36.262300 (+ 3983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.262311 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.262332 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.263009 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.263448 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.263456 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.263462 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.263843 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.264721 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:36.264732 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:36.265332 (+ 600us) client_negotiation.cc:769] Sending connection context
0812 11:23:36.265752 (+ 420us) client_negotiation.cc:240] Negotiation successful
0812 11:23:36.266455 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":360,"threads_started":1}
I20260812 11:23:36.300113 15018 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:36811
I20260812 11:23:36.300132 15236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:36811 every 8 connection(s)
I20260812 11:23:36.301523 15018 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:23:36.301878 15058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.283505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:38779 (local address 127.12.110.254:36401)
0812 11:23:36.283747 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:36.283755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:36.284707 (+ 952us) server_negotiation.cc:407] Connection header received
0812 11:23:36.284971 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:36.284978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:36.285082 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:36.285204 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:36.287028 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.287850 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.289318 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.289754 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.289928 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.293657 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:36.293663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:36.293666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:36.293700 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:36.297282 (+ 3582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.298845 (+ 1563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.298855 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.298862 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.299193 (+ 331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.299985 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.299993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.299997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.300499 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:36.300675 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:36.301177 (+ 502us) server_negotiation.cc:299] Negotiation successful
0812 11:23:36.301441 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260812 11:23:36.302028 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.283690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36401 (local address 127.12.110.194:38779)
0812 11:23:36.284584 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:36.284603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:36.284795 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:36.285275 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:36.285286 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:36.286161 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:36.286818 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.286837 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.288016 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.288021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.289142 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.289155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.289981 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.289988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.290180 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.291080 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:36.291114 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:36.293266 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:36.297520 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.297532 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.297540 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.298447 (+ 907us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.299565 (+ 1118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.299570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.299573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.299744 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.300650 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:36.300656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:36.300878 (+ 222us) client_negotiation.cc:769] Sending connection context
0812 11:23:36.301041 (+ 163us) client_negotiation.cc:240] Negotiation successful
0812 11:23:36.301449 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":551,"threads_started":1}
I20260812 11:23:36.302194 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15018
I20260812 11:23:36.302469 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:23:36.303203 12731 external_mini_cluster.cc:1428] Setting key a711d5ba64f6624d3aa264dd00d0d913
I20260812 11:23:36.305364 15238 heartbeater.cc:347] Connected to a master server at 127.12.110.254:36401
I20260812 11:23:36.305801 15238 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:36.306900 15238 heartbeater.cc:511] Master 127.12.110.254:36401 requested a full tablet report, sending...
I20260812 11:23:36.308766 14666 ts_manager.cc:194] Registered new tserver with Master: b403c8f308b047f48b505f98ce992a46 (127.12.110.194:36811)
I20260812 11:23:36.309851 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:52777
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:23:36.367057 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36401
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:23:36.729442 15013 heartbeater.cc:503] Master 127.12.110.254:36401 was elected leader, sending a full tablet report...
W20260812 11:23:36.748872 15248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:36.749311 15248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:36.749572 15248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:36.758545 15248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:36.758685 15248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:36.758863 15248 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:23:36.758957 15248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:23:36.771123 15248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:36401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:36.772989 15248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:36.775152 15248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:36.789997 15257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:36.790194 15259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:36.791204 15261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:36.791947 15248 server_base.cc:1034] running on GCE node
I20260812 11:23:36.792591 15248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:36.793730 15248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:36.795228 15248 hybrid_clock.cc:648] HybridClock initialized: now 1786533816795040 us; error 102 us; skew 500 ppm
I20260812 11:23:36.795642 15248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:36 dist-test-slave-h6n0 krb5kdc[14591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:36.799670 15248 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:23:36.802469 15248 webserver.cc:460] Webserver started at http://127.12.110.195:37483/ using document root <none> and password file <none>
I20260812 11:23:36.803244 15248 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:36.803375 15248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:36.804078 15248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:36.808544 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "9a83c441c5ae4ca7ae24c60e485c09e0"
format_stamp: "Formatted at 2026-08-12 11:23:36 on dist-test-slave-h6n0"
server_key: "f8ce710194e76bb30ba587e284f3c2d0"
server_key_iv: "98a118badb9183a7f1772b79bfbce34b"
server_key_version: "encryptionkey@0"
I20260812 11:23:36.810128 15248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "9a83c441c5ae4ca7ae24c60e485c09e0"
format_stamp: "Formatted at 2026-08-12 11:23:36 on dist-test-slave-h6n0"
server_key: "f8ce710194e76bb30ba587e284f3c2d0"
server_key_iv: "98a118badb9183a7f1772b79bfbce34b"
server_key_version: "encryptionkey@0"
I20260812 11:23:36.817569 15248 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260812 11:23:36.822842 15273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:36.825418 15248 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260812 11:23:36.825711 15248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "9a83c441c5ae4ca7ae24c60e485c09e0"
format_stamp: "Formatted at 2026-08-12 11:23:36 on dist-test-slave-h6n0"
server_key: "f8ce710194e76bb30ba587e284f3c2d0"
server_key_iv: "98a118badb9183a7f1772b79bfbce34b"
server_key_version: "encryptionkey@0"
I20260812 11:23:36.826030 15248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:36.871029 15248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:36.875785 15248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:36.876138 15248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:36.879561 15248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:36.879668 15248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:36.879801 15248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:36.879884 15248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:23:36 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:36.910707 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.880258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:59183 (local address 127.12.110.254:36401)
0812 11:23:36.880941 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:36.880945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:36.882459 (+ 1514us) server_negotiation.cc:407] Connection header received
0812 11:23:36.885564 (+ 3105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:36.885571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:36.885696 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:36.886044 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:36.889006 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.890019 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.892838 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.893731 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.894108 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.900709 (+ 6601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:36.900715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:36.900719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:36.900761 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:36.905491 (+ 4730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.906845 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.906853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.906859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.907042 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.908078 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.908084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.908089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.908512 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:36.908752 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:36.909811 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0812 11:23:36.910213 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":419,"threads_started":1}
I20260812 11:23:36.911952 15289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.880921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36401 (local address 127.12.110.195:59183)
0812 11:23:36.882132 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:36.882246 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:36.884932 (+ 2686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:36.886345 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:36.886363 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:36.887497 (+ 1134us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:36.888749 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.888785 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.890216 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.890224 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.892670 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.892684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.894220 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.894232 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.894663 (+ 431us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.895991 (+ 1328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:36.896029 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:36.900289 (+ 4260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:36.905742 (+ 5453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.905761 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.905781 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.906629 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.907217 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.907225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.907232 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.907749 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.908716 (+ 967us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:36.908727 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:36.909297 (+ 570us) client_negotiation.cc:769] Sending connection context
0812 11:23:36.909670 (+ 373us) client_negotiation.cc:240] Negotiation successful
0812 11:23:36.910584 (+ 914us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":865,"thread_start_us":506,"threads_started":1}
I20260812 11:23:36.940007 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.920020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:52037 (local address 127.12.110.254:36401)
0812 11:23:36.920201 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:36.920207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:36.921901 (+ 1694us) server_negotiation.cc:407] Connection header received
0812 11:23:36.922228 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:36.922235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:36.922340 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:36.922438 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:36.924525 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.925528 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.928204 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.928845 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.928988 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.932788 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:36.932793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:36.932797 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:36.932842 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:36.936664 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.937619 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.937625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.937628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.937810 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:36.938356 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:36.938363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:36.938369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:36.938749 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:36.938930 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:36.939378 (+ 448us) server_negotiation.cc:299] Negotiation successful
0812 11:23:36.939625 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:23:36.940109 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:36.920289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36401 (local address 127.12.110.195:52037)
0812 11:23:36.921770 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:36.921798 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:36.922033 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:36.922780 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:36.922784 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:36.923698 (+ 914us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:36.924326 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.924348 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.925763 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.925771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.927947 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:36.927968 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:36.929462 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:36.929484 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:36.929685 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:36.930752 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:36.930801 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:36.932548 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:36.936810 (+ 4262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.936816 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.936823 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.937475 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.937956 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:36.937966 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:36.937972 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:36.938198 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:36.938884 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:36.938891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:36.939094 (+ 203us) client_negotiation.cc:769] Sending connection context
0812 11:23:36.939249 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 11:23:36.939623 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1279,"thread_start_us":623,"threads_started":1}
I20260812 11:23:36.947654 15248 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:39575
I20260812 11:23:36.947701 15466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:39575 every 8 connection(s)
I20260812 11:23:36.949508 15248 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:23:36.952099 15467 heartbeater.cc:347] Connected to a master server at 127.12.110.254:36401
I20260812 11:23:36.952396 15467 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:36.953145 15467 heartbeater.cc:511] Master 127.12.110.254:36401 requested a full tablet report, sending...
I20260812 11:23:36.955015 14666 ts_manager.cc:194] Registered new tserver with Master: 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195:39575)
I20260812 11:23:36.956050 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:59183
I20260812 11:23:36.956784 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15248
I20260812 11:23:36.957028 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:23:36.957643 12731 external_mini_cluster.cc:1428] Setting key d2e45b2bbecd4199218fadc8aed9e8fa
I20260812 11:23:36.961684 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:23:36 dist-test-slave-h6n0 krb5kdc[14591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:23:37 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:37.025166 15288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.008376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53562 (local address 127.12.110.254:36401)
0812 11:23:37.008590 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.008600 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.009023 (+ 423us) server_negotiation.cc:407] Connection header received
0812 11:23:37.009275 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.009279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.009345 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.009462 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:37.011000 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.011786 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.013089 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.013493 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.013633 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.019250 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:37.019256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:37.019259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:37.019296 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:37.022050 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:37.022852 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:37.022856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:37.022859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:37.023033 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:37.023778 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:37.023782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:37.023785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:37.024081 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:37.024293 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.024716 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.024884 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:23:37.031299 14666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:23:37.039851 14666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:37.067394 15488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.055397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:39685 (local address 127.0.0.1:34602)
0812 11:23:37.057749 (+ 2352us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.057770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.057992 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.058964 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.058969 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.058995 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.059481 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.059495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.061451 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.061459 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.064195 (+ 2736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.064214 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.065986 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.065991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.066245 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.066305 (+ 60us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.066475 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.066719 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2130,"mutex_wait_us":1,"thread_start_us":384,"threads_started":1}
I20260812 11:23:37.067435 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.055759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34602 (local address 127.12.110.193:39685)
0812 11:23:37.057078 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.057086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.057867 (+ 781us) server_negotiation.cc:407] Connection header received
0812 11:23:37.058183 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.058192 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.058398 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.058624 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.059789 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.061291 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.064375 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.065781 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.065987 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.066183 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.066594 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.066778 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":888,"threads_started":1}
I20260812 11:23:37.067891 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.054552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:36811 (local address 127.0.0.1:58926)
0812 11:23:37.056176 (+ 1624us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.056220 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.056396 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.057697 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.057708 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.057773 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.058691 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.058709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.060999 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.061011 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.064781 (+ 3770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.064795 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.066909 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.066917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.067105 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.067143 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.067414 (+ 271us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.067570 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":2681,"threads_started":1}
I20260812 11:23:37.068835 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.054757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.12.110.194:36811)
0812 11:23:37.056951 (+ 2194us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.056962 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.056997 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:23:37.057109 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.057119 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.057415 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.058932 (+ 1517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.059162 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.060705 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.064974 (+ 4269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.066369 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.066621 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.066785 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.068124 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.068511 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1879,"thread_start_us":1727,"threads_started":1}
I20260812 11:23:37.072518 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.059206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.12.110.195:39575)
0812 11:23:37.060183 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.060196 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.062763 (+ 2567us) server_negotiation.cc:407] Connection header received
0812 11:23:37.062962 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.062973 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.063208 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.063410 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.064496 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.065705 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.069531 (+ 3826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.070618 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.070831 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.071434 (+ 603us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.071778 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.071998 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":538,"threads_started":1}
I20260812 11:23:37.072686 14896 tablet_service.cc:1467] Processing CreateTablet for tablet 245abf5fe94b4640a811de98bc0a3bb5 (DEFAULT_TABLE table=test-table [id=414ac365c65c47eeaf74d95260546bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:37.072692 15125 tablet_service.cc:1467] Processing CreateTablet for tablet 245abf5fe94b4640a811de98bc0a3bb5 (DEFAULT_TABLE table=test-table [id=414ac365c65c47eeaf74d95260546bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:37.073408 15496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.057782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:39575 (local address 127.0.0.1:44576)
0812 11:23:37.062595 (+ 4813us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.062617 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.062800 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.063697 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.063702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.063733 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.064143 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.064154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.065850 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.065856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.069336 (+ 3480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.069354 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.070823 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.070829 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.071451 (+ 622us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.071508 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.071695 (+ 187us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.071905 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4629,"thread_start_us":4425,"threads_started":1}
I20260812 11:23:37.074288 14896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245abf5fe94b4640a811de98bc0a3bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.074291 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245abf5fe94b4640a811de98bc0a3bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.077095 15350 tablet_service.cc:1467] Processing CreateTablet for tablet 245abf5fe94b4640a811de98bc0a3bb5 (DEFAULT_TABLE table=test-table [id=414ac365c65c47eeaf74d95260546bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:37.078212 15350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 245abf5fe94b4640a811de98bc0a3bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.089659 15501 tablet_bootstrap.cc:492] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Bootstrap starting.
I20260812 11:23:37.090627 15499 tablet_bootstrap.cc:492] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Bootstrap starting.
I20260812 11:23:37.095515 15502 tablet_bootstrap.cc:492] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Bootstrap starting.
I20260812 11:23:37.096108 15501 tablet_bootstrap.cc:654] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.096670 15499 tablet_bootstrap.cc:654] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.097507 15501 log.cc:824] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:37.097688 15499 log.cc:824] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:37.099021 15502 tablet_bootstrap.cc:654] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.100279 15502 log.cc:824] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:37.101118 15501 tablet_bootstrap.cc:492] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: No bootstrap required, opened a new log
I20260812 11:23:37.101444 15501 ts_tablet_manager.cc:1397] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260812 11:23:37.102002 15499 tablet_bootstrap.cc:492] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: No bootstrap required, opened a new log
I20260812 11:23:37.102270 15499 ts_tablet_manager.cc:1397] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260812 11:23:37.103082 15502 tablet_bootstrap.cc:492] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: No bootstrap required, opened a new log
I20260812 11:23:37.103336 15502 ts_tablet_manager.cc:1397] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:23:37.113034 15502 raft_consensus.cc:348] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.113482 15502 raft_consensus.cc:374] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.113580 15502 raft_consensus.cc:729] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a83c441c5ae4ca7ae24c60e485c09e0, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.114313 15502 consensus_queue.cc:260] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.115477 15501 raft_consensus.cc:348] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.116080 15467 heartbeater.cc:503] Master 127.12.110.254:36401 was elected leader, sending a full tablet report...
I20260812 11:23:37.116108 15501 raft_consensus.cc:374] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.116217 15501 raft_consensus.cc:729] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b403c8f308b047f48b505f98ce992a46, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.116818 15502 ts_tablet_manager.cc:1428] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260812 11:23:37.117110 15501 consensus_queue.cc:260] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.115475 15499 raft_consensus.cc:348] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.118400 15238 heartbeater.cc:503] Master 127.12.110.254:36401 was elected leader, sending a full tablet report...
I20260812 11:23:37.118999 15501 ts_tablet_manager.cc:1428] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Time spent starting tablet: real 0.017s user 0.018s sys 0.001s
I20260812 11:23:37.119637 15499 raft_consensus.cc:374] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.119793 15499 raft_consensus.cc:729] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babd9b5f948441d8ba0df35635c86400, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.121107 15499 consensus_queue.cc:260] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.122901 15509 raft_consensus.cc:482] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:37.123167 15499 ts_tablet_manager.cc:1428] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Time spent starting tablet: real 0.021s user 0.014s sys 0.005s
I20260812 11:23:37.123334 15509 raft_consensus.cc:504] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.125195 15509 leader_election.cc:288] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b403c8f308b047f48b505f98ce992a46 (127.12.110.194:36811), 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195:39575)
I20260812 11:23:37.136030 15489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.125569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:59991 (local address 127.12.110.194:36811)
0812 11:23:37.125750 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.125755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.127066 (+ 1311us) server_negotiation.cc:407] Connection header received
0812 11:23:37.127413 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.127418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.127488 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.127544 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.128388 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.129605 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.132848 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.134757 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.134952 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.135053 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.135608 (+ 555us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.135760 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:23:37.136401 15510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.125802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:36811 (local address 127.12.110.193:59991)
0812 11:23:37.126915 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.126938 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.127213 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.127736 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.127740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.127774 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.128216 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.128226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.129814 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.129824 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.132623 (+ 2799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.132637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.135058 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.135065 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.135266 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.135308 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.135590 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.135927 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":3,"thread_start_us":596,"threads_started":1}
I20260812 11:23:37.137540 15161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abf5fe94b4640a811de98bc0a3bb5" candidate_uuid: "babd9b5f948441d8ba0df35635c86400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b403c8f308b047f48b505f98ce992a46" is_pre_election: true
I20260812 11:23:37.137979 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.127381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:57191 (local address 127.12.110.195:39575)
0812 11:23:37.127608 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.127615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.128661 (+ 1046us) server_negotiation.cc:407] Connection header received
0812 11:23:37.128961 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.128967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.129050 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.129173 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.130273 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.131663 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.134863 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.136555 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.136762 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.136870 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.137508 (+ 638us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.137664 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:23:37.138062 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.127017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:39575 (local address 127.12.110.193:57191)
0812 11:23:37.128499 (+ 1482us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.128530 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.128762 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.129364 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.129369 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.129395 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.130018 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.130032 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.131904 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.131910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.134635 (+ 2725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.134654 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.137048 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.137056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.137263 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.137301 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.137427 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.137602 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":34,"thread_start_us":1958,"threads_started":1}
I20260812 11:23:37.138180 15161 raft_consensus.cc:2454] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babd9b5f948441d8ba0df35635c86400 in term 0.
I20260812 11:23:37.139133 14811 leader_election.cc:302] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b403c8f308b047f48b505f98ce992a46, babd9b5f948441d8ba0df35635c86400; no voters:
I20260812 11:23:37.139376 15391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abf5fe94b4640a811de98bc0a3bb5" candidate_uuid: "babd9b5f948441d8ba0df35635c86400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" is_pre_election: true
I20260812 11:23:37.139731 15509 raft_consensus.cc:2781] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:37.139981 15509 raft_consensus.cc:482] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:37.139990 15391 raft_consensus.cc:2454] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate babd9b5f948441d8ba0df35635c86400 in term 0.
I20260812 11:23:37.140178 15509 raft_consensus.cc:3037] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.143352 15509 raft_consensus.cc:504] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.144032 15509 leader_election.cc:288] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [CANDIDATE]: Term 1 election: Requested vote from peers b403c8f308b047f48b505f98ce992a46 (127.12.110.194:36811), 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195:39575)
I20260812 11:23:37.144668 15161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abf5fe94b4640a811de98bc0a3bb5" candidate_uuid: "babd9b5f948441d8ba0df35635c86400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b403c8f308b047f48b505f98ce992a46"
I20260812 11:23:37.144843 15391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "245abf5fe94b4640a811de98bc0a3bb5" candidate_uuid: "babd9b5f948441d8ba0df35635c86400" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0"
I20260812 11:23:37.144906 15161 raft_consensus.cc:3037] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.145078 15391 raft_consensus.cc:3037] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.149091 15391 raft_consensus.cc:2454] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babd9b5f948441d8ba0df35635c86400 in term 1.
I20260812 11:23:37.149087 15161 raft_consensus.cc:2454] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate babd9b5f948441d8ba0df35635c86400 in term 1.
I20260812 11:23:37.149605 14811 leader_election.cc:302] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b403c8f308b047f48b505f98ce992a46, babd9b5f948441d8ba0df35635c86400; no voters:
I20260812 11:23:37.150005 15509 raft_consensus.cc:2781] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:37.150854 15509 raft_consensus.cc:686] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 1 LEADER]: Becoming Leader. State: Replica: babd9b5f948441d8ba0df35635c86400, State: Running, Role: LEADER
I20260812 11:23:37.151597 15509 consensus_queue.cc:237] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } }
I20260812 11:23:37.157473 14665 catalog_manager.cc:5495] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 reported cstate change: term changed from 0 to 1, leader changed from <none> to babd9b5f948441d8ba0df35635c86400 (127.12.110.193). New cstate: current_term: 1 leader_uuid: "babd9b5f948441d8ba0df35635c86400" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } health_report { overall_health: HEALTHY } } }
W20260812 11:23:37.201538 15468 tablet.cc:2369] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:23:37.203007 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.196808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34616 (local address 127.12.110.193:39685)
0812 11:23:37.197100 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.197106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.197131 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:23:37.197417 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.197422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.197533 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.197676 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:37.198113 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.198935 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.200334 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.200750 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.200844 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.201429 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.202147 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.202323 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.202451 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.202646 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 11:23:37.211982 15161 raft_consensus.cc:1261] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Refusing update from remote peer babd9b5f948441d8ba0df35635c86400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:37.211989 15391 raft_consensus.cc:1261] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Refusing update from remote peer babd9b5f948441d8ba0df35635c86400: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:37.213164 15512 consensus_queue.cc:1035] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:37.213899 15509 consensus_queue.cc:1035] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:23:37.220191 15015 tablet.cc:2369] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:23:37.227193 15521 mvcc.cc:205] Tried to move back new op lower bound from 7317642515289382912 to 7317642515057483776. Current Snapshot: MvccSnapshot[applied={T|T < 7317642515289382912}]
I20260812 11:23:37.229833 15519 mvcc.cc:205] Tried to move back new op lower bound from 7317642515289382912 to 7317642515057483776. Current Snapshot: MvccSnapshot[applied={T|T < 7317642515289382912 or (T in {7317642515289382912})}]
I20260812 11:23:37.247251 14665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:37.247562 14665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:37.252334 14665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728: Sending DeleteTablet for 3 replicas of tablet 245abf5fe94b4640a811de98bc0a3bb5
I20260812 11:23:37.253954 15125 tablet_service.cc:1514] Processing DeleteTablet for tablet 245abf5fe94b4640a811de98bc0a3bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:37 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:58926
I20260812 11:23:37.254063 14896 tablet_service.cc:1514] Processing DeleteTablet for tablet 245abf5fe94b4640a811de98bc0a3bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:37 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:34602
I20260812 11:23:37.254823 15350 tablet_service.cc:1514] Processing DeleteTablet for tablet 245abf5fe94b4640a811de98bc0a3bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:37 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:44576
I20260812 11:23:37.255090 15534 tablet_replica.cc:330] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: stopping tablet replica
I20260812 11:23:37.255369 15535 tablet_replica.cc:330] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: stopping tablet replica
I20260812 11:23:37.255636 15534 raft_consensus.cc:2229] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:37.255767 15535 raft_consensus.cc:2229] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:23:37.255811 14665 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:23:37.256408 15535 raft_consensus.cc:2258] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:37.256433 15534 raft_consensus.cc:2258] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260812 11:23:37.256497 14665 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:37.258701 15535 ts_tablet_manager.cc:1905] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:37.259050 15534 ts_tablet_manager.cc:1905] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:37.266206 15534 ts_tablet_manager.cc:1918] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:23:37.266472 15534 log.cc:1196] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/245abf5fe94b4640a811de98bc0a3bb5
I20260812 11:23:37.267083 15125 tablet_service.cc:1467] Processing CreateTablet for tablet dbe28af97cc149e3bb040c14e5795a09 (DEFAULT_TABLE table=test-table [id=cd786057ef804369bd68c3f2aa362d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:37.267226 15534 ts_tablet_manager.cc:1939] T 245abf5fe94b4640a811de98bc0a3bb5 P b403c8f308b047f48b505f98ce992a46: Deleting consensus metadata
I20260812 11:23:37.267594 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe28af97cc149e3bb040c14e5795a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.268164 15535 ts_tablet_manager.cc:1918] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:23:37.268356 15535 log.cc:1196] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/245abf5fe94b4640a811de98bc0a3bb5
I20260812 11:23:37.268918 15535 ts_tablet_manager.cc:1939] T 245abf5fe94b4640a811de98bc0a3bb5 P babd9b5f948441d8ba0df35635c86400: Deleting consensus metadata
I20260812 11:23:37.270666 14641 catalog_manager.cc:4841] TS b403c8f308b047f48b505f98ce992a46 (127.12.110.194:36811): tablet 245abf5fe94b4640a811de98bc0a3bb5 (table test-table [id=414ac365c65c47eeaf74d95260546bfa]) successfully deleted
I20260812 11:23:37.270711 14643 catalog_manager.cc:4841] TS babd9b5f948441d8ba0df35635c86400 (127.12.110.193:39685): tablet 245abf5fe94b4640a811de98bc0a3bb5 (table test-table [id=414ac365c65c47eeaf74d95260546bfa]) successfully deleted
I20260812 11:23:37.271207 14896 tablet_service.cc:1467] Processing CreateTablet for tablet dbe28af97cc149e3bb040c14e5795a09 (DEFAULT_TABLE table=test-table [id=cd786057ef804369bd68c3f2aa362d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:37.271659 14896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe28af97cc149e3bb040c14e5795a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.271369 15348 tablet_service.cc:1467] Processing CreateTablet for tablet dbe28af97cc149e3bb040c14e5795a09 (DEFAULT_TABLE table=test-table [id=cd786057ef804369bd68c3f2aa362d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:37.272152 15348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe28af97cc149e3bb040c14e5795a09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.274425 15501 tablet_bootstrap.cc:492] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Bootstrap starting.
I20260812 11:23:37.276111 15501 tablet_bootstrap.cc:654] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.279511 15499 tablet_bootstrap.cc:492] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: Bootstrap starting.
I20260812 11:23:37.279712 15502 tablet_bootstrap.cc:492] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: Bootstrap starting.
I20260812 11:23:37.281219 15502 tablet_bootstrap.cc:654] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.281248 15499 tablet_bootstrap.cc:654] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.290896 15536 tablet_replica.cc:330] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: stopping tablet replica
I20260812 11:23:37.290964 15502 tablet_bootstrap.cc:492] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: No bootstrap required, opened a new log
I20260812 11:23:37.291414 15502 ts_tablet_manager.cc:1397] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260812 11:23:37.291942 15536 raft_consensus.cc:2229] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:37.292865 15536 raft_consensus.cc:2258] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:37.293274 15502 raft_consensus.cc:348] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.293601 15502 raft_consensus.cc:374] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.293757 15502 raft_consensus.cc:729] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a83c441c5ae4ca7ae24c60e485c09e0, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.293895 15501 tablet_bootstrap.cc:492] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: No bootstrap required, opened a new log
I20260812 11:23:37.294051 15501 ts_tablet_manager.cc:1397] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.008s
I20260812 11:23:37.294255 15502 consensus_queue.cc:260] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.294997 15501 raft_consensus.cc:348] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.295217 15501 raft_consensus.cc:374] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.295318 15501 raft_consensus.cc:729] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b403c8f308b047f48b505f98ce992a46, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.295539 15499 tablet_bootstrap.cc:492] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: No bootstrap required, opened a new log
I20260812 11:23:37.295840 15499 ts_tablet_manager.cc:1397] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260812 11:23:37.296247 15502 ts_tablet_manager.cc:1428] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 11:23:37.296761 15499 raft_consensus.cc:348] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.296996 15499 raft_consensus.cc:374] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.297111 15499 raft_consensus.cc:729] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babd9b5f948441d8ba0df35635c86400, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.297740 15499 consensus_queue.cc:260] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.297709 15501 consensus_queue.cc:260] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.298354 15499 ts_tablet_manager.cc:1428] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260812 11:23:37.300143 15536 ts_tablet_manager.cc:1905] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:37.304139 15501 ts_tablet_manager.cc:1428] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260812 11:23:37.307243 15536 ts_tablet_manager.cc:1918] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:23:37.307427 15536 log.cc:1196] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/245abf5fe94b4640a811de98bc0a3bb5
I20260812 11:23:37.307981 15536 ts_tablet_manager.cc:1939] T 245abf5fe94b4640a811de98bc0a3bb5 P 9a83c441c5ae4ca7ae24c60e485c09e0: Deleting consensus metadata
I20260812 11:23:37.309827 14641 catalog_manager.cc:4841] TS 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195:39575): tablet 245abf5fe94b4640a811de98bc0a3bb5 (table test-table [id=414ac365c65c47eeaf74d95260546bfa]) successfully deleted
W20260812 11:23:37.310041 15239 tablet.cc:2369] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:23:37.469762 15508 raft_consensus.cc:482] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:37.470068 15508 raft_consensus.cc:504] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.472110 15508 leader_election.cc:288] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers babd9b5f948441d8ba0df35635c86400 (127.12.110.193:39685), 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195:39575)
I20260812 11:23:37.480952 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.472716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:40529 (local address 127.12.110.195:39575)
0812 11:23:37.472923 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.472929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.474288 (+ 1359us) server_negotiation.cc:407] Connection header received
0812 11:23:37.474450 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.474455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.474575 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.474673 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.475489 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.476655 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.479172 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.479969 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.480163 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.480248 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.480554 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.480688 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:23:37.481082 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.472494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:39575 (local address 127.12.110.194:40529)
0812 11:23:37.474162 (+ 1668us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.474181 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.474313 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.474719 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.474729 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.474750 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.475210 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.475220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.476893 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.476898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.479013 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.479023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.480200 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.480206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.480374 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.480398 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.480505 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.480632 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1534,"mutex_wait_us":8,"thread_start_us":1359,"threads_started":1}
I20260812 11:23:37.481736 15391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe28af97cc149e3bb040c14e5795a09" candidate_uuid: "b403c8f308b047f48b505f98ce992a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" is_pre_election: true
I20260812 11:23:37.481984 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.473085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:33097 (local address 127.12.110.193:39685)
0812 11:23:37.473270 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.473276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.473852 (+ 576us) server_negotiation.cc:407] Connection header received
0812 11:23:37.474232 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.474240 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.474328 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.474421 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.475306 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.476286 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.479029 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.480434 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.480596 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.480663 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.481511 (+ 848us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.481659 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:23:37.482016 15391 raft_consensus.cc:2454] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b403c8f308b047f48b505f98ce992a46 in term 0.
I20260812 11:23:37.482188 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.472455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:39685 (local address 127.12.110.194:33097)
0812 11:23:37.473728 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.473750 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.473986 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.474463 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.474467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.474510 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.475049 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.475065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.476436 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.476439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.478802 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.478818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.480834 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.480843 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.481010 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.481359 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.481458 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.481592 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1142,"mutex_wait_us":78,"thread_start_us":964,"threads_started":1}
I20260812 11:23:37.483129 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe28af97cc149e3bb040c14e5795a09" candidate_uuid: "b403c8f308b047f48b505f98ce992a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babd9b5f948441d8ba0df35635c86400" is_pre_election: true
I20260812 11:23:37.483654 14927 raft_consensus.cc:2454] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b403c8f308b047f48b505f98ce992a46 in term 0.
I20260812 11:23:37.483842 15046 leader_election.cc:302] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a83c441c5ae4ca7ae24c60e485c09e0, b403c8f308b047f48b505f98ce992a46; no voters:
I20260812 11:23:37.484280 15508 raft_consensus.cc:2781] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:37.484422 15508 raft_consensus.cc:482] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:37.484539 15508 raft_consensus.cc:3037] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.486927 15508 raft_consensus.cc:504] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.487633 15508 leader_election.cc:288] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [CANDIDATE]: Term 1 election: Requested vote from peers babd9b5f948441d8ba0df35635c86400 (127.12.110.193:39685), 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195:39575)
I20260812 11:23:37.487986 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe28af97cc149e3bb040c14e5795a09" candidate_uuid: "b403c8f308b047f48b505f98ce992a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babd9b5f948441d8ba0df35635c86400"
I20260812 11:23:37.488258 14927 raft_consensus.cc:3037] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.488402 15391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe28af97cc149e3bb040c14e5795a09" candidate_uuid: "b403c8f308b047f48b505f98ce992a46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0"
I20260812 11:23:37.488649 15391 raft_consensus.cc:3037] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.491638 15391 raft_consensus.cc:2454] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b403c8f308b047f48b505f98ce992a46 in term 1.
I20260812 11:23:37.491904 14927 raft_consensus.cc:2454] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b403c8f308b047f48b505f98ce992a46 in term 1.
I20260812 11:23:37.492056 15046 leader_election.cc:302] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a83c441c5ae4ca7ae24c60e485c09e0, b403c8f308b047f48b505f98ce992a46; no voters:
I20260812 11:23:37.492350 15508 raft_consensus.cc:2781] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:37.492867 15508 raft_consensus.cc:686] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 1 LEADER]: Becoming Leader. State: Replica: b403c8f308b047f48b505f98ce992a46, State: Running, Role: LEADER
I20260812 11:23:37.493274 15508 consensus_queue.cc:237] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.496949 14666 catalog_manager.cc:5495] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 reported cstate change: term changed from 0 to 1, leader changed from <none> to b403c8f308b047f48b505f98ce992a46 (127.12.110.194). New cstate: current_term: 1 leader_uuid: "b403c8f308b047f48b505f98ce992a46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } health_report { overall_health: UNKNOWN } } }
Aug 12 11:23:37 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533814, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:37.580484 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.557311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36401 (local address 127.0.0.1:53568)
0812 11:23:37.558254 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.558276 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.558447 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.558804 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.558808 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.559572 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:37.559865 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.559874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.560969 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.560977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.561925 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.561937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.562766 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.562773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.562963 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.569229 (+ 6266us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:37.569263 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:37.572489 (+ 3226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:37.576921 (+ 4432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:37.576944 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:37.576959 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:37.577459 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:37.578046 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:37.578053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:37.578060 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:37.578318 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:37.579197 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:37.579209 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:37.579452 (+ 243us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.579652 (+ 200us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.580013 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":597,"threads_started":1}
I20260812 11:23:37.580547 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.557285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53568 (local address 127.12.110.254:36401)
0812 11:23:37.558052 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.558064 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.558363 (+ 299us) server_negotiation.cc:407] Connection header received
0812 11:23:37.558598 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.558603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.558656 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.558776 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:37.560043 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.560839 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.562074 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.562587 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.562722 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.572789 (+ 10067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:37.572795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:37.572799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:37.572834 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:37.576740 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:37.577602 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:37.577617 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:37.577626 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:37.577815 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:37.578635 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:37.578641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:37.578643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:37.579006 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:37.579164 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.579765 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.580211 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":231,"server-negotiator.queue_time_us":566,"thread_start_us":356,"threads_started":1}
I20260812 11:23:37.587092 14666 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:53568:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:23:37.588661 14666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:37.596508 15125 tablet_service.cc:1467] Processing CreateTablet for tablet c061e873d1fd4bd49161ab519e148f8c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111d33f8c4ed4409957ee5bafefd0714]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:23:37.596515 14896 tablet_service.cc:1467] Processing CreateTablet for tablet c061e873d1fd4bd49161ab519e148f8c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111d33f8c4ed4409957ee5bafefd0714]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:23:37.596741 15350 tablet_service.cc:1467] Processing CreateTablet for tablet c061e873d1fd4bd49161ab519e148f8c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=111d33f8c4ed4409957ee5bafefd0714]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:23:37.597113 14896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c061e873d1fd4bd49161ab519e148f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.597108 15125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c061e873d1fd4bd49161ab519e148f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.597345 15350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c061e873d1fd4bd49161ab519e148f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:37.605268 15501 tablet_bootstrap.cc:492] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: Bootstrap starting.
I20260812 11:23:37.605607 15502 tablet_bootstrap.cc:492] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: Bootstrap starting.
I20260812 11:23:37.606932 15499 tablet_bootstrap.cc:492] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: Bootstrap starting.
I20260812 11:23:37.607267 15501 tablet_bootstrap.cc:654] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.609242 15499 tablet_bootstrap.cc:654] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.609990 15501 tablet_bootstrap.cc:492] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: No bootstrap required, opened a new log
I20260812 11:23:37.610208 15501 ts_tablet_manager.cc:1397] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:23:37.611159 15501 raft_consensus.cc:348] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.611449 15501 raft_consensus.cc:374] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.611543 15501 raft_consensus.cc:729] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b403c8f308b047f48b505f98ce992a46, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.611881 15501 consensus_queue.cc:260] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.612433 15502 tablet_bootstrap.cc:654] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:37.612447 15508 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.612797 15508 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:37.613867 15501 ts_tablet_manager.cc:1428] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 11:23:37.617565 15502 tablet_bootstrap.cc:492] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: No bootstrap required, opened a new log
I20260812 11:23:37.617762 15502 ts_tablet_manager.cc:1397] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260812 11:23:37.618644 15502 raft_consensus.cc:348] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.618916 15502 raft_consensus.cc:374] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.619012 15502 raft_consensus.cc:729] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a83c441c5ae4ca7ae24c60e485c09e0, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.619352 15502 consensus_queue.cc:260] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.619856 15507 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.620196 15507 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:37.621395 15499 tablet_bootstrap.cc:492] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: No bootstrap required, opened a new log
I20260812 11:23:37.621609 15499 ts_tablet_manager.cc:1397] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20260812 11:23:37.622439 15499 raft_consensus.cc:348] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.622869 15499 raft_consensus.cc:374] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:37.622977 15502 ts_tablet_manager.cc:1428] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:23:37.622978 15499 raft_consensus.cc:729] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: babd9b5f948441d8ba0df35635c86400, State: Initialized, Role: FOLLOWER
I20260812 11:23:37.623281 15499 consensus_queue.cc:260] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.624006 15499 ts_tablet_manager.cc:1428] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260812 11:23:37.623787 15512 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.624204 15512 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:37.818516 15507 raft_consensus.cc:482] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:37.818990 15507 raft_consensus.cc:504] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.820937 15507 leader_election.cc:288] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers babd9b5f948441d8ba0df35635c86400 (127.12.110.193:39685), b403c8f308b047f48b505f98ce992a46 (127.12.110.194:36811)
I20260812 11:23:37.827401 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.821251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:54391 (local address 127.12.110.193:39685)
0812 11:23:37.821766 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.821776 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.821813 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:23:37.822164 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.822169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.822234 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.822353 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.822993 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.823698 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.825512 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.826409 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.826642 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.826721 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.826990 (+ 269us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.827124 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334}
I20260812 11:23:37.827486 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.821107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:39685 (local address 127.12.110.195:54391)
0812 11:23:37.821711 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.821737 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.821901 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.822391 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.822397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.822438 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.822801 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.822813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.823858 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.823867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.825341 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.825354 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.826628 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.826634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.826812 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.826848 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.826963 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.827102 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":3,"spinlock_wait_cycles":2304,"thread_start_us":976,"threads_started":1}
I20260812 11:23:37.827656 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.821392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:38145 (local address 127.12.110.194:36811)
0812 11:23:37.821998 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.822005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.822447 (+ 442us) server_negotiation.cc:407] Connection header received
0812 11:23:37.822742 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.822746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.822805 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.822913 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:37.823377 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.824081 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.825843 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.826668 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.826848 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.826922 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.827197 (+ 275us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.827331 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"spinlock_wait_cycles":3840,"thread_start_us":429,"threads_started":1}
I20260812 11:23:37.828118 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.821168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:36811 (local address 127.12.110.195:38145)
0812 11:23:37.822335 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.822350 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.822580 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.822929 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.822933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.822957 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:37.823231 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.823238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.824223 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.824228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.825689 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.825701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.826873 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.826879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.827037 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.827063 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.827170 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.827299 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"mutex_wait_us":65,"spinlock_wait_cycles":7296,"thread_start_us":308,"threads_started":1}
I20260812 11:23:37.828127 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061e873d1fd4bd49161ab519e148f8c" candidate_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babd9b5f948441d8ba0df35635c86400" is_pre_election: true
I20260812 11:23:37.828373 14927 raft_consensus.cc:2454] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a83c441c5ae4ca7ae24c60e485c09e0 in term 0.
I20260812 11:23:37.828842 15161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061e873d1fd4bd49161ab519e148f8c" candidate_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b403c8f308b047f48b505f98ce992a46" is_pre_election: true
I20260812 11:23:37.829125 15161 raft_consensus.cc:2454] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9a83c441c5ae4ca7ae24c60e485c09e0 in term 0.
I20260812 11:23:37.829219 15277 leader_election.cc:302] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a83c441c5ae4ca7ae24c60e485c09e0, babd9b5f948441d8ba0df35635c86400; no voters:
I20260812 11:23:37.830029 15507 raft_consensus.cc:2781] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:37.830219 15507 raft_consensus.cc:482] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:37.830325 15507 raft_consensus.cc:3037] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.833518 15507 raft_consensus.cc:504] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.834273 15507 leader_election.cc:288] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [CANDIDATE]: Term 1 election: Requested vote from peers babd9b5f948441d8ba0df35635c86400 (127.12.110.193:39685), b403c8f308b047f48b505f98ce992a46 (127.12.110.194:36811)
I20260812 11:23:37.834800 14927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061e873d1fd4bd49161ab519e148f8c" candidate_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "babd9b5f948441d8ba0df35635c86400"
I20260812 11:23:37.835026 14927 raft_consensus.cc:3037] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.835143 15161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c061e873d1fd4bd49161ab519e148f8c" candidate_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b403c8f308b047f48b505f98ce992a46"
I20260812 11:23:37.835404 15161 raft_consensus.cc:3037] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:37.837157 14927 raft_consensus.cc:2454] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a83c441c5ae4ca7ae24c60e485c09e0 in term 1.
I20260812 11:23:37.837581 15277 leader_election.cc:302] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a83c441c5ae4ca7ae24c60e485c09e0, babd9b5f948441d8ba0df35635c86400; no voters:
I20260812 11:23:37.837823 15161 raft_consensus.cc:2454] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9a83c441c5ae4ca7ae24c60e485c09e0 in term 1.
I20260812 11:23:37.837965 15507 raft_consensus.cc:2781] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:37.838485 15507 raft_consensus.cc:686] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 LEADER]: Becoming Leader. State: Replica: 9a83c441c5ae4ca7ae24c60e485c09e0, State: Running, Role: LEADER
I20260812 11:23:37.838907 15507 consensus_queue.cc:237] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } }
I20260812 11:23:37.840767 15572 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: TxnStatusTablet state changed. Reason: New leader 9a83c441c5ae4ca7ae24c60e485c09e0. Latest consensus state: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.841099 15572 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:23:37.841972 15576 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:23:37.842286 15576 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:23:37.842988 14665 catalog_manager.cc:5495] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195). New cstate: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } health_report { overall_health: HEALTHY } } }
I20260812 11:23:37.861356 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.854960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:39575 (local address 127.0.0.1:44590)
0812 11:23:37.855161 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.855181 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.855303 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.855807 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.855811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.855821 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:37.855979 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.855986 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.857497 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.857501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.858693 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.858706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.859790 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.859795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.859927 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.859979 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.860930 (+ 951us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.860946 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.861011 (+ 65us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.861132 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 11:23:37.861424 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.854993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.12.110.195:39575)
0812 11:23:37.855182 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.855188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.855241 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:23:37.855486 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.855493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.855602 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.855760 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:37.856143 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.857333 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.858833 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.859585 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.859716 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.860176 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.860815 (+ 639us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.860960 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.861047 (+ 87us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.861127 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:23:37.866250 15161 raft_consensus.cc:1261] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Refusing update from remote peer 9a83c441c5ae4ca7ae24c60e485c09e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:37.866609 14927 raft_consensus.cc:1261] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Refusing update from remote peer 9a83c441c5ae4ca7ae24c60e485c09e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:37.866911 15507 consensus_queue.cc:1035] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:37.867347 15572 consensus_queue.cc:1035] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:37.870945 15512 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: TxnStatusTablet state changed. Reason: New leader 9a83c441c5ae4ca7ae24c60e485c09e0. Latest consensus state: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.871261 15512 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:37.873481 15582 mvcc.cc:205] Tried to move back new op lower bound from 7317642517973716992 to 7317642517872103424. Current Snapshot: MvccSnapshot[applied={T|T < 7317642517973716992}]
I20260812 11:23:37.877739 15507 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.878033 15507 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:23:37.884351 15543 consensus_queue.cc:1035] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:37.885882 15512 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.886219 15512 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P babd9b5f948441d8ba0df35635c86400: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:37.899849 15508 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: TxnStatusTablet state changed. Reason: New leader 9a83c441c5ae4ca7ae24c60e485c09e0. Latest consensus state: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.900174 15508 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:37.905844 15572 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.906116 15572 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P 9a83c441c5ae4ca7ae24c60e485c09e0: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:23:37.906901 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.890024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.12.110.194:36811)
0812 11:23:37.890364 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.890371 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.899179 (+ 8808us) server_negotiation.cc:407] Connection header received
0812 11:23:37.899534 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.899541 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.899667 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.899800 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:37.900578 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.901676 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.903755 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.904414 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.904515 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.905194 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.905907 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.906016 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.906299 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.906411 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260812 11:23:37.907176 15543 consensus_queue.cc:1035] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:37.926237 15508 tablet_replica.cc:439] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "babd9b5f948441d8ba0df35635c86400" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39685 } } peers { permanent_uuid: "b403c8f308b047f48b505f98ce992a46" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36811 } } peers { permanent_uuid: "9a83c441c5ae4ca7ae24c60e485c09e0" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 39575 } } }
I20260812 11:23:37.926631 15508 tablet_replica.cc:442] T c061e873d1fd4bd49161ab519e148f8c P b403c8f308b047f48b505f98ce992a46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:37.936985 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.929738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:50673 (local address 127.12.110.195:39575)
0812 11:23:37.929977 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.929984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.930601 (+ 617us) server_negotiation.cc:407] Connection header received
0812 11:23:37.930816 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.930821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.930935 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.931062 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:37.931664 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.932500 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.934597 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.935418 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.935519 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.936140 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.936301 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.936385 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.936559 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.936624 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:23:37.937078 15603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.929747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:39575 (local address 127.12.110.194:50673)
0812 11:23:37.930447 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.930468 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.930668 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.931091 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.931095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.931108 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:37.931491 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.931503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.932655 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.932660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.934402 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.934418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.935609 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.935617 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.935790 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.935858 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.936420 (+ 562us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.936433 (+ 13us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.936522 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.936664 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":387,"threads_started":1}
I20260812 11:23:37.978277 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.970683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:57167 (local address 127.12.110.194:36811)
0812 11:23:37.970862 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:37.970869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:37.971723 (+ 854us) server_negotiation.cc:407] Connection header received
0812 11:23:37.971919 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:37.971926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:37.972023 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:37.972122 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:37.973336 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.974124 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.975995 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.976774 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.976928 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.977459 (+ 531us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.977592 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.977700 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:37.977850 (+ 150us) server_negotiation.cc:299] Negotiation successful
0812 11:23:37.977931 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:23:37.979044 15607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:37.970422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:36811 (local address 127.12.110.195:57167)
0812 11:23:37.971596 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:37.971616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:37.971774 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:37.972281 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:37.972287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:37.972300 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:37.973168 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.973178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.974263 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.974270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.975842 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:37.975856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:37.977011 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:37.977019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:37.977205 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:37.977260 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:37.977702 (+ 442us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:37.977717 (+ 15us) client_negotiation.cc:769] Sending connection context
0812 11:23:37.977830 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:23:37.977971 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":1696,"threads_started":1}
I20260812 11:23:38.051946 14665 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:38.052285 14665 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:38.056607 14665 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4f7ea935c8e5425184dce26661ef8728: Sending DeleteTablet for 3 replicas of tablet dbe28af97cc149e3bb040c14e5795a09
I20260812 11:23:38.057705 15125 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe28af97cc149e3bb040c14e5795a09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:38 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:58926
I20260812 11:23:38.057713 14896 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe28af97cc149e3bb040c14e5795a09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:38 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:34602
I20260812 11:23:38.058506 15615 tablet_replica.cc:330] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: stopping tablet replica
I20260812 11:23:38.058521 15350 tablet_service.cc:1514] Processing DeleteTablet for tablet dbe28af97cc149e3bb040c14e5795a09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:38 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:44576
I20260812 11:23:38.058907 15615 raft_consensus.cc:2229] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:38.059499 15615 raft_consensus.cc:2258] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:38.061229 15615 ts_tablet_manager.cc:1905] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:38.071045 15615 ts_tablet_manager.cc:1918] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:38.071286 15615 log.cc:1196] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/dbe28af97cc149e3bb040c14e5795a09
I20260812 11:23:38.071906 15615 ts_tablet_manager.cc:1939] T dbe28af97cc149e3bb040c14e5795a09 P babd9b5f948441d8ba0df35635c86400: Deleting consensus metadata
I20260812 11:23:38.073367 14643 catalog_manager.cc:4841] TS babd9b5f948441d8ba0df35635c86400 (127.12.110.193:39685): tablet dbe28af97cc149e3bb040c14e5795a09 (table test-table [id=cd786057ef804369bd68c3f2aa362d6c]) successfully deleted
I20260812 11:23:38.079057 15613 tablet_replica.cc:330] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: stopping tablet replica
I20260812 11:23:38.079233 15617 tablet_replica.cc:330] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: stopping tablet replica
I20260812 11:23:38.079283 15613 raft_consensus.cc:2229] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:23:38.079464 15617 raft_consensus.cc:2229] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:38.079769 15617 raft_consensus.cc:2258] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:38.079964 15613 raft_consensus.cc:2258] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:38.081161 15613 ts_tablet_manager.cc:1905] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:38.081279 15617 ts_tablet_manager.cc:1905] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:38.089982 15613 ts_tablet_manager.cc:1918] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:38.090142 15613 log.cc:1196] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/dbe28af97cc149e3bb040c14e5795a09
I20260812 11:23:38.090724 15613 ts_tablet_manager.cc:1939] T dbe28af97cc149e3bb040c14e5795a09 P b403c8f308b047f48b505f98ce992a46: Deleting consensus metadata
I20260812 11:23:38.091958 14641 catalog_manager.cc:4841] TS b403c8f308b047f48b505f98ce992a46 (127.12.110.194:36811): tablet dbe28af97cc149e3bb040c14e5795a09 (table test-table [id=cd786057ef804369bd68c3f2aa362d6c]) successfully deleted
I20260812 11:23:38.092064 15617 ts_tablet_manager.cc:1918] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:38.092227 15617 log.cc:1196] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/dbe28af97cc149e3bb040c14e5795a09
I20260812 11:23:38.092747 15617 ts_tablet_manager.cc:1939] T dbe28af97cc149e3bb040c14e5795a09 P 9a83c441c5ae4ca7ae24c60e485c09e0: Deleting consensus metadata
I20260812 11:23:38.093833 14641 catalog_manager.cc:4841] TS 9a83c441c5ae4ca7ae24c60e485c09e0 (127.12.110.195:39575): tablet dbe28af97cc149e3bb040c14e5795a09 (table test-table [id=cd786057ef804369bd68c3f2aa362d6c]) successfully deleted
Aug 12 11:23:38 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:38.114202 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:38.086856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34618 (local address 127.12.110.193:39685)
0812 11:23:38.087082 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:38.087089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:38.094559 (+ 7470us) server_negotiation.cc:407] Connection header received
0812 11:23:38.094776 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:38.094782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:38.094885 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:38.095001 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:38.097199 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.098323 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.100174 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.100853 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.100991 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:38.106516 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:38.106560 (+ 44us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:38.106582 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:38.106629 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:38.110773 (+ 4144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.111711 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.111721 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.111724 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.111885 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.112426 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.112439 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.112443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.112875 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:38.113264 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:38.113579 (+ 315us) server_negotiation.cc:299] Negotiation successful
0812 11:23:38.113822 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260812 11:23:38.115078 15005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34618
I20260812 11:23:38.144161 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:38.118791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.12.110.254:36401)
0812 11:23:38.126007 (+ 7216us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:38.126015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:38.126783 (+ 768us) server_negotiation.cc:407] Connection header received
0812 11:23:38.126996 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:38.127003 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:38.127092 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:38.127214 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:38.129216 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.130735 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.132832 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.133574 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.133760 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:38.137196 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:38.137204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:38.137210 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:38.137260 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:38.140998 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.141846 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.141852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.141857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.142006 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.142437 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.142443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.142447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.142855 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:38.143120 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:38.143515 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 11:23:38.143748 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7063,"thread_start_us":6902,"threads_started":1}
W20260812 11:23:38.144773 14665 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53570
I20260812 11:23:38.147456 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14785
I20260812 11:23:38.165596 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:38.149716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.12.110.193:39685)
0812 11:23:38.149910 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:38.149915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:38.150453 (+ 538us) server_negotiation.cc:407] Connection header received
0812 11:23:38.150636 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:38.150642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:38.150743 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:38.150840 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:38.152283 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.153330 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.155167 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.155802 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.155898 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:38.158957 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:38.158963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:38.158968 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:38.159005 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:38.162461 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.163288 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.163294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.163297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.163461 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.163944 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.163950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.163954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.164320 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:38.164591 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:38.164990 (+ 399us) server_negotiation.cc:299] Negotiation successful
0812 11:23:38.165231 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 11:23:38.166163 15005 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34628
W20260812 11:23:38.166795 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14785: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:38.167152 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14785
I20260812 11:23:38.229194 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15018
Aug 12 11:23:38 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:23:38.255710 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:38.230204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58944 (local address 127.12.110.194:36811)
0812 11:23:38.230406 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:38.230412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:38.230610 (+ 198us) server_negotiation.cc:407] Connection header received
0812 11:23:38.230754 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:38.230759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:38.230872 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:38.230986 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:38.232802 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.234465 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.236969 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.237909 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.238081 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:38.245193 (+ 7112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:38.245212 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:38.245235 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:38.245288 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:38.250294 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.251499 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.251512 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.251515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.251841 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.252486 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.252492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.252496 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.253431 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:38.253918 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:38.254293 (+ 375us) server_negotiation.cc:299] Negotiation successful
0812 11:23:38.254685 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 11:23:38.256796 15231 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58944
W20260812 11:23:38.257786 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15018: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:38.257972 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15018
I20260812 11:23:38.299705 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15248
Aug 12 11:23:38 dist-test-slave-h6n0 krb5kdc[14591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:23:38.320766 15498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:38.300528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44596 (local address 127.12.110.195:39575)
0812 11:23:38.300714 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:38.300718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:38.300805 (+ 87us) server_negotiation.cc:407] Connection header received
0812 11:23:38.300993 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:38.300997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:38.301064 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:38.301152 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:38.303066 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.304138 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.306137 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:38.306763 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:38.306877 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:38.313167 (+ 6290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:38.313187 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:38.313208 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:38.313267 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:38.317283 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.318229 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.318238 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.318241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.318409 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:38.318869 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:38.318885 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:38.318888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:38.319344 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:38.319745 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:38.319997 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 11:23:38.320252 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 11:23:38.321744 15459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44596
W20260812 11:23:38.322608 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15248: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:38.322818 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15248
I20260812 11:23:38.356968 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14611
I20260812 11:23:38.358259 14731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:38.498306 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14611
2026-08-12T11:23:38Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5699 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:38 dist-test-slave-h6n0 krb5kdc[15632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:38 dist-test-slave-h6n0 krb5kdc[15632](info): set up 2 sockets
Aug 12 11:23:38 dist-test-slave-h6n0 krb5kdc[15632](info): commencing operation
krb5kdc: starting...
W20260812 11:23:39.917274 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.359s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:23:40 dist-test-slave-h6n0 krb5kdc[15632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:23:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:23:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:23:40.191558 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:32865
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:32865
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:40.580150 15652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:40.580536 15652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:40.580719 15652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:40.589143 15652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:23:40.589331 15652 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:23:40.589449 15652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:40.589571 15652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:40.589671 15652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:23:40.589776 15652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:23:40.600301 15652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:32865
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:32865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:40.601687 15652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:40.603119 15652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:40.615463 15661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:40.615743 15662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:40.615868 15664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:40.616710 15652 server_base.cc:1034] running on GCE node
I20260812 11:23:40.617584 15652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:40.619130 15652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:40.620365 15652 hybrid_clock.cc:648] HybridClock initialized: now 1786533820620288 us; error 63 us; skew 500 ppm
I20260812 11:23:40.620714 15652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:40 dist-test-slave-h6n0 krb5kdc[15632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533820, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:40.624271 15652 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:23:40.625945 15652 webserver.cc:460] Webserver started at http://127.12.110.254:33595/ using document root <none> and password file <none>
I20260812 11:23:40.626677 15652 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:40.626775 15652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:40.627038 15652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:40.629981 15652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "394a5f95769a44488fc15a94828b61d1"
format_stamp: "Formatted at 2026-08-12 11:23:40 on dist-test-slave-h6n0"
server_key: "3f1913c84fba03d1a2c34576fa4ad835"
server_key_iv: "c71522f8dfc172e4c03cebd91e6b49df"
server_key_version: "encryptionkey@0"
I20260812 11:23:40.630810 15652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "394a5f95769a44488fc15a94828b61d1"
format_stamp: "Formatted at 2026-08-12 11:23:40 on dist-test-slave-h6n0"
server_key: "3f1913c84fba03d1a2c34576fa4ad835"
server_key_iv: "c71522f8dfc172e4c03cebd91e6b49df"
server_key_version: "encryptionkey@0"
I20260812 11:23:40.635993 15652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260812 11:23:40.639657 15678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:40.641218 15652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:23:40.641422 15652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "394a5f95769a44488fc15a94828b61d1"
format_stamp: "Formatted at 2026-08-12 11:23:40 on dist-test-slave-h6n0"
server_key: "3f1913c84fba03d1a2c34576fa4ad835"
server_key_iv: "c71522f8dfc172e4c03cebd91e6b49df"
server_key_version: "encryptionkey@0"
I20260812 11:23:40.641662 15652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:40.673089 15652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:40.679263 15652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:40.679602 15652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:40.705107 15652 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:32865
I20260812 11:23:40.705137 15782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:32865 every 8 connection(s)
I20260812 11:23:40.706575 15652 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:23:40.710500 15783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:40.710831 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15652
I20260812 11:23:40.711135 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:23:40.711987 12731 external_mini_cluster.cc:1428] Setting key 153339e2659029fb88e96f5cd060f21f
I20260812 11:23:40.730057 15783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Bootstrap starting.
Aug 12 11:23:40 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:40.733857 15783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:40.735164 15783 log.cc:824] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:40.738811 15783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: No bootstrap required, opened a new log
I20260812 11:23:40.745510 15787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:40.714363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.12.110.254:32865)
0812 11:23:40.715581 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:40.715606 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:40.715685 (+ 79us) server_negotiation.cc:407] Connection header received
0812 11:23:40.719196 (+ 3511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:40.719246 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:40.720072 (+ 826us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:40.720948 (+ 876us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:40.722566 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:40.724693 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:40.726230 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:40.728020 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:40.728260 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:40.732881 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:40.732896 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:40.732919 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:40.732976 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:40.738017 (+ 5041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:40.739243 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:40.739253 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:40.739261 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:40.739535 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:40.740406 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:40.740415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:40.740422 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:40.741490 (+ 1068us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:40.741968 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:40.742411 (+ 443us) server_negotiation.cc:299] Negotiation successful
0812 11:23:40.743209 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":519,"threads_started":1}
I20260812 11:23:40.750852 15783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394a5f95769a44488fc15a94828b61d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32865 } }
I20260812 11:23:40.751297 15783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:40.751406 15783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394a5f95769a44488fc15a94828b61d1, State: Initialized, Role: FOLLOWER
I20260812 11:23:40.752034 15783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394a5f95769a44488fc15a94828b61d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32865 } }
I20260812 11:23:40.752429 15783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:23:40.752604 15783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:23:40.752835 15783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:40.755446 15783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394a5f95769a44488fc15a94828b61d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32865 } }
I20260812 11:23:40.756045 15783 leader_election.cc:302] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394a5f95769a44488fc15a94828b61d1; no voters:
I20260812 11:23:40.756835 15783 leader_election.cc:288] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:23:40.757269 15790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:40.758322 15790 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [term 1 LEADER]: Becoming Leader. State: Replica: 394a5f95769a44488fc15a94828b61d1, State: Running, Role: LEADER
I20260812 11:23:40.759620 15783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:23:40.759373 15790 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394a5f95769a44488fc15a94828b61d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32865 } }
I20260812 11:23:40.762910 15795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394a5f95769a44488fc15a94828b61d1. Latest consensus state: current_term: 1 leader_uuid: "394a5f95769a44488fc15a94828b61d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394a5f95769a44488fc15a94828b61d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32865 } } }
I20260812 11:23:40.763695 15795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:40.763471 15793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "394a5f95769a44488fc15a94828b61d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394a5f95769a44488fc15a94828b61d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32865 } } }
I20260812 11:23:40.763988 15793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:40.765549 15802 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:23:40.773109 15802 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:23:40.782403 15802 catalog_manager.cc:1347] Generated new cluster ID: de721b6cc2a14cb5896055f3ac02036b
I20260812 11:23:40.782588 15802 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:23:40.832612 15802 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:23:40.834105 15802 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:23:40.852419 15802 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Generated new TSK 0
I20260812 11:23:40.853446 15802 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:23:40.926816 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32865
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:41.295979 15825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:41.296365 15825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:41.296562 15825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:41.304188 15825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:41.304352 15825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:41.304514 15825 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:23:41.304606 15825 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:23:41.304745 15825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:23:41.315400 15825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:32865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:41.316808 15825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:41.318192 15825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:41.332263 15834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:41.332381 15836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:41.332356 15839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:41.333385 15825 server_base.cc:1034] running on GCE node
I20260812 11:23:41.334205 15825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:41.335300 15825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:41.336617 15825 hybrid_clock.cc:648] HybridClock initialized: now 1786533821336528 us; error 42 us; skew 500 ppm
I20260812 11:23:41.337172 15825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:41 dist-test-slave-h6n0 krb5kdc[15632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533821, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:41.341074 15825 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:23:41.342684 15825 webserver.cc:460] Webserver started at http://127.12.110.193:35215/ using document root <none> and password file <none>
I20260812 11:23:41.343336 15825 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:41.343446 15825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:41.343714 15825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:41.346989 15825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "804fa5988396483ea6e6f1d786c7c34a"
format_stamp: "Formatted at 2026-08-12 11:23:41 on dist-test-slave-h6n0"
server_key: "d62708a55688d49b3aee05b4f80cbdac"
server_key_iv: "523072a45724d3c5fd242eebfde8248b"
server_key_version: "encryptionkey@0"
I20260812 11:23:41.347724 15825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "804fa5988396483ea6e6f1d786c7c34a"
format_stamp: "Formatted at 2026-08-12 11:23:41 on dist-test-slave-h6n0"
server_key: "d62708a55688d49b3aee05b4f80cbdac"
server_key_iv: "523072a45724d3c5fd242eebfde8248b"
server_key_version: "encryptionkey@0"
I20260812 11:23:41.353075 15825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 11:23:41.357172 15851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:41.359166 15825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:23:41.359445 15825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "804fa5988396483ea6e6f1d786c7c34a"
format_stamp: "Formatted at 2026-08-12 11:23:41 on dist-test-slave-h6n0"
server_key: "d62708a55688d49b3aee05b4f80cbdac"
server_key_iv: "523072a45724d3c5fd242eebfde8248b"
server_key_version: "encryptionkey@0"
I20260812 11:23:41.359697 15825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:41.384694 15825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:41.390723 15825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:41.391002 15825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:41.394306 15825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:41.394418 15825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:41.394620 15825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:41.394716 15825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:23:41 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533821, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:41.420392 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:41.395584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:38689 (local address 127.12.110.254:32865)
0812 11:23:41.396260 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:41.396268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:41.397740 (+ 1472us) server_negotiation.cc:407] Connection header received
0812 11:23:41.401603 (+ 3863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:41.401612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:41.401701 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:41.401840 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:41.404118 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.404910 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.406573 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.407275 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.407426 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:41.412742 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:41.412748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:41.412751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:41.412785 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:41.416479 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:41.417600 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:41.417608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:41.417612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:41.417745 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:41.418212 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:41.418217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:41.418220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:41.418502 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:41.418720 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:41.419932 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0812 11:23:41.420120 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":328,"threads_started":1}
I20260812 11:23:41.422469 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:41.396417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32865 (local address 127.12.110.193:38689)
0812 11:23:41.397498 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:41.397562 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:41.400937 (+ 3375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:41.402136 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:41.402144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:41.402976 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:41.403893 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:41.403925 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.405081 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.405087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:41.406399 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:41.406412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.407781 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.407792 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:41.408153 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:41.409080 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:41.409104 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:41.412459 (+ 3355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:41.416706 (+ 4247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:41.416715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:41.416732 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:41.417448 (+ 716us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:41.417863 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:41.417867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:41.417871 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:41.418087 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:41.418715 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:41.418727 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:41.419373 (+ 646us) client_negotiation.cc:769] Sending connection context
0812 11:23:41.419792 (+ 419us) client_negotiation.cc:240] Negotiation successful
0812 11:23:41.420545 (+ 753us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":405,"threads_started":1}
I20260812 11:23:41.450153 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:41.430262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:50641 (local address 127.12.110.254:32865)
0812 11:23:41.430588 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:41.430600 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:41.432002 (+ 1402us) server_negotiation.cc:407] Connection header received
0812 11:23:41.432288 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:41.432299 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:41.432421 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:41.432560 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:41.434379 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.435460 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.437709 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.438740 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.438933 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:41.442080 (+ 3147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:41.442086 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:41.442091 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:41.442126 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:41.445605 (+ 3479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:41.446363 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:41.446368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:41.446372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:41.446597 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:41.447173 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:41.447180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:41.447187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:41.447649 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:41.447811 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:41.449393 (+ 1582us) server_negotiation.cc:299] Negotiation successful
0812 11:23:41.449706 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 11:23:41.450377 15952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:41.430182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32865 (local address 127.12.110.193:50641)
0812 11:23:41.431826 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:41.431859 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:41.432075 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:41.432600 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:41.432611 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:41.433623 (+ 1012us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:41.434206 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:41.434223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.435613 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.435622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:41.437532 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:41.437556 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:41.439100 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:41.439115 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:41.439435 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:41.440255 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:41.440288 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:41.441806 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:41.445760 (+ 3954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:41.445771 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:41.445774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:41.446233 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:41.446760 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:41.446768 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:41.446774 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:41.446978 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:41.447911 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:41.447917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:41.448669 (+ 752us) client_negotiation.cc:769] Sending connection context
0812 11:23:41.449084 (+ 415us) client_negotiation.cc:240] Negotiation successful
0812 11:23:41.449660 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"thread_start_us":1020,"threads_started":1}
I20260812 11:23:41.462829 15825 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:46809
I20260812 11:23:41.462862 16036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:46809 every 8 connection(s)
I20260812 11:23:41.464128 15825 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:23:41.465905 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15825
I20260812 11:23:41.466156 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:23:41.466666 16037 heartbeater.cc:347] Connected to a master server at 127.12.110.254:32865
I20260812 11:23:41.466921 16037 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:41.467250 12731 external_mini_cluster.cc:1428] Setting key fc0d228f7ca2feb110c42f9ed2269786
I20260812 11:23:41.467759 16037 heartbeater.cc:511] Master 127.12.110.254:32865 requested a full tablet report, sending...
I20260812 11:23:41.470211 15708 ts_manager.cc:194] Registered new tserver with Master: 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809)
I20260812 11:23:41.472570 15708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:38689
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:23:41.521330 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32865
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:41.907668 16043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:41.907953 16043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:41.908147 16043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:41.917008 16043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:41.917124 16043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:41.917317 16043 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:23:41.917382 16043 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:23:41.917503 16043 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:23:41.930292 16043 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:32865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:41.931663 16043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:41.932955 16043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:41.947198 16051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:41.947448 16053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:41.947901 16050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:41.948822 16043 server_base.cc:1034] running on GCE node
I20260812 11:23:41.949530 16043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:41.951328 16043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:41.952863 16043 hybrid_clock.cc:648] HybridClock initialized: now 1786533821952609 us; error 123 us; skew 500 ppm
I20260812 11:23:41.953351 16043 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:41 dist-test-slave-h6n0 krb5kdc[15632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533821, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:41.957883 16043 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:23:41.959688 16043 webserver.cc:460] Webserver started at http://127.12.110.194:46355/ using document root <none> and password file <none>
I20260812 11:23:41.960440 16043 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:41.960588 16043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:41.960934 16043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:41.965106 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "f6e0647c24294aa68b67596506bb05fd"
format_stamp: "Formatted at 2026-08-12 11:23:41 on dist-test-slave-h6n0"
server_key: "21fd817771361765263e72d4d019c00d"
server_key_iv: "08b408e0e00e39a0b7ce3d650c946ce3"
server_key_version: "encryptionkey@0"
I20260812 11:23:41.966300 16043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "f6e0647c24294aa68b67596506bb05fd"
format_stamp: "Formatted at 2026-08-12 11:23:41 on dist-test-slave-h6n0"
server_key: "21fd817771361765263e72d4d019c00d"
server_key_iv: "08b408e0e00e39a0b7ce3d650c946ce3"
server_key_version: "encryptionkey@0"
I20260812 11:23:41.973992 16043 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260812 11:23:41.977875 16067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:41.979902 16043 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:23:41.980314 16043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "f6e0647c24294aa68b67596506bb05fd"
format_stamp: "Formatted at 2026-08-12 11:23:41 on dist-test-slave-h6n0"
server_key: "21fd817771361765263e72d4d019c00d"
server_key_iv: "08b408e0e00e39a0b7ce3d650c946ce3"
server_key_version: "encryptionkey@0"
I20260812 11:23:41.980607 16043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:42.002439 16043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:42.007777 16043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:42.008129 16043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:42.011267 16043 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:42.011368 16043 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:42.011497 16043 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:42.011590 16043 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:23:42 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533821, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:42.041358 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.012535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:58925 (local address 127.12.110.254:32865)
0812 11:23:42.013731 (+ 1196us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.013740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.014815 (+ 1075us) server_negotiation.cc:407] Connection header received
0812 11:23:42.018871 (+ 4056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.018880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.018998 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.019106 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:42.021792 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.023015 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.025403 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.026036 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.026181 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.032015 (+ 5834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:42.032022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:42.032026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:42.032075 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:42.036590 (+ 4515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.037727 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.037737 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.037742 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.037933 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.038604 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.038610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.038614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.039093 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:42.039252 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.040623 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.040876 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1053,"thread_start_us":940,"threads_started":1}
I20260812 11:23:42.043540 16086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.013542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32865 (local address 127.12.110.194:58925)
0812 11:23:42.014621 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.014673 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.018160 (+ 3487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.019763 (+ 1603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.019772 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.020613 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:42.021608 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.021630 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.023174 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.023179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.025225 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.025246 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.026313 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.026321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.026671 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.027478 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:42.027516 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:42.031533 (+ 4017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:42.036914 (+ 5381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.036922 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.036940 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.037487 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.038062 (+ 575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.038068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.038072 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.038437 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.039454 (+ 1017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:42.039466 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:42.040095 (+ 629us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.040522 (+ 427us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.041160 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":461,"threads_started":1}
I20260812 11:23:42.062790 16043 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:37455
I20260812 11:23:42.062882 16277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:37455 every 8 connection(s)
I20260812 11:23:42.064615 16043 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:23:42.067516 16278 heartbeater.cc:347] Connected to a master server at 127.12.110.254:32865
I20260812 11:23:42.067822 16278 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:42.069034 16278 heartbeater.cc:511] Master 127.12.110.254:32865 requested a full tablet report, sending...
I20260812 11:23:42.070423 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16043
I20260812 11:23:42.070698 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:23:42.071235 12731 external_mini_cluster.cc:1428] Setting key 0bd7ab5d5b1c3d4f0c1458fefa33ea27
I20260812 11:23:42.072788 15708 ts_manager.cc:194] Registered new tserver with Master: f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455)
I20260812 11:23:42.073901 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.056526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32865 (local address 127.12.110.194:55807)
0812 11:23:42.057758 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.057775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.057980 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.058334 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.058339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.058953 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:42.059662 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.059687 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.060789 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.060795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.062642 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.062664 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.063675 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.063680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.063849 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.064626 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:42.064644 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:42.065946 (+ 1302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:42.070455 (+ 4509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.070460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.070463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.071141 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.071652 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.071657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.071661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.071859 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.072759 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:42.072767 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:42.072971 (+ 204us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.073117 (+ 146us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.073460 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"thread_start_us":754,"threads_started":1}
I20260812 11:23:42.073902 16084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.056284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:55807 (local address 127.12.110.254:32865)
0812 11:23:42.056487 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.056494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.057893 (+ 1399us) server_negotiation.cc:407] Connection header received
0812 11:23:42.058114 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.058119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.058199 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.058308 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:42.059824 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.060623 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.062850 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.063376 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.063452 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.066190 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:42.066196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:42.066203 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:42.066241 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:42.070265 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.071296 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.071301 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.071304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.071492 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.072021 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.072027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.072031 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.072601 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:42.072811 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.073272 (+ 461us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.073554 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:23:42.073907 15708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:58925
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:23:42.131225 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32865
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:23:42.476243 16037 heartbeater.cc:503] Master 127.12.110.254:32865 was elected leader, sending a full tablet report...
W20260812 11:23:42.506450 16283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:42.506996 16283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:42.507206 16283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:42.515961 16283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:42.516166 16283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:42.516350 16283 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260812 11:23:42.516445 16283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:23:42.516597 16283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:23:42.527315 16283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:32865
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:42.528887 16283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:42.530241 16283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:42.545114 16294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:42.545493 16293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:42.545794 16296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:42.546164 16283 server_base.cc:1034] running on GCE node
I20260812 11:23:42.546974 16283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:42.548331 16283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:42.549574 16283 hybrid_clock.cc:648] HybridClock initialized: now 1786533822549506 us; error 66 us; skew 500 ppm
I20260812 11:23:42.549966 16283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:42 dist-test-slave-h6n0 krb5kdc[15632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533822, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:42.553751 16283 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:23:42.555634 16283 webserver.cc:460] Webserver started at http://127.12.110.195:37841/ using document root <none> and password file <none>
I20260812 11:23:42.556504 16283 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:42.556629 16283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:42.557145 16283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:42.560504 16283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "74ea12fe16814693bc4588364d7e7b8f"
format_stamp: "Formatted at 2026-08-12 11:23:42 on dist-test-slave-h6n0"
server_key: "ec668979c6c4f5cbb941f9f4178bfa98"
server_key_iv: "ea7d66e3b2c3c795e1114f205eb74ea5"
server_key_version: "encryptionkey@0"
I20260812 11:23:42.561415 16283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "74ea12fe16814693bc4588364d7e7b8f"
format_stamp: "Formatted at 2026-08-12 11:23:42 on dist-test-slave-h6n0"
server_key: "ec668979c6c4f5cbb941f9f4178bfa98"
server_key_iv: "ea7d66e3b2c3c795e1114f205eb74ea5"
server_key_version: "encryptionkey@0"
I20260812 11:23:42.568040 16283 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260812 11:23:42.576689 16305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:42.578692 16283 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.003s
I20260812 11:23:42.578886 16283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "74ea12fe16814693bc4588364d7e7b8f"
format_stamp: "Formatted at 2026-08-12 11:23:42 on dist-test-slave-h6n0"
server_key: "ec668979c6c4f5cbb941f9f4178bfa98"
server_key_iv: "ea7d66e3b2c3c795e1114f205eb74ea5"
server_key_version: "encryptionkey@0"
I20260812 11:23:42.579183 16283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:42.604964 16283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:42.611651 16283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:42.612005 16283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:42.615989 16283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:42.616153 16283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:42.616400 16283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:42.616509 16283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:23:42 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533822, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:42.643105 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.615996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:48581 (local address 127.12.110.254:32865)
0812 11:23:42.616490 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.616497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.617814 (+ 1317us) server_negotiation.cc:407] Connection header received
0812 11:23:42.620001 (+ 2187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.620010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.620100 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.620237 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:42.622225 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.623297 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.625830 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.626552 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.626687 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.634637 (+ 7950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:42.634643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:42.634647 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:42.634687 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:42.639122 (+ 4435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.640235 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.640242 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.640246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.640400 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.641031 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.641038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.641044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.641481 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:42.641641 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.642402 (+ 761us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.642696 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
I20260812 11:23:42.644136 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.616679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32865 (local address 127.12.110.195:48581)
0812 11:23:42.617620 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.617678 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.619651 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.620431 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.620439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.621042 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:42.622013 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.622040 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.623482 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.623487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.625677 (+ 2190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.625694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.626850 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.626862 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.627228 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.628648 (+ 1420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:42.628707 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:42.634314 (+ 5607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:42.639360 (+ 5046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.639372 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.639396 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.640103 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.640581 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.640588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.640594 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.640893 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.641636 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:42.641644 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:42.642035 (+ 391us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.642310 (+ 275us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.642922 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":427,"threads_started":1}
I20260812 11:23:42.672468 16431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.653805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32865 (local address 127.12.110.195:60823)
0812 11:23:42.655145 (+ 1340us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.655172 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.655375 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.655867 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.655871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.656442 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:42.656762 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.656772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.657952 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.657959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.659419 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.659433 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.660566 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.660574 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.660808 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.661732 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:42.661750 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:42.662515 (+ 765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:42.667825 (+ 5310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.667839 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.667844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.668829 (+ 985us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.669601 (+ 772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:42.669607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:42.669612 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:42.669855 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:42.670748 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:42.670753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:42.671020 (+ 267us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.671199 (+ 179us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.671751 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1138,"thread_start_us":958,"threads_started":1}
I20260812 11:23:42.672871 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.653822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:60823 (local address 127.12.110.254:32865)
0812 11:23:42.654091 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.654102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.655318 (+ 1216us) server_negotiation.cc:407] Connection header received
0812 11:23:42.655504 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.655508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.655605 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.655840 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:42.656923 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.657806 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.659583 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.660364 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.660493 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.662845 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:42.662854 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:42.662862 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:42.662908 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:42.667506 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.669193 (+ 1687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.669204 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.669210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.669406 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.670033 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.670041 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.670046 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.670494 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:42.670678 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.671924 (+ 1246us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.672424 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 11:23:42.681619 16283 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:41709
I20260812 11:23:42.681671 16502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:41709 every 8 connection(s)
I20260812 11:23:42.683408 16283 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:23:42.686133 16506 heartbeater.cc:347] Connected to a master server at 127.12.110.254:32865
I20260812 11:23:42.686743 16506 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:42.687812 16506 heartbeater.cc:511] Master 127.12.110.254:32865 requested a full tablet report, sending...
I20260812 11:23:42.689870 15708 ts_manager.cc:194] Registered new tserver with Master: 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709)
I20260812 11:23:42.690893 15708 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:48581
I20260812 11:23:42.692565 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16283
I20260812 11:23:42.692833 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:23:42.693698 12731 external_mini_cluster.cc:1428] Setting key c64ca353eceedfe1936bd3de3da1d0b2
I20260812 11:23:42.698045 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:23:42 dist-test-slave-h6n0 krb5kdc[15632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:23:42 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:42.769626 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.752484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.12.110.254:32865)
0812 11:23:42.752807 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.752816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.752975 (+ 159us) server_negotiation.cc:407] Connection header received
0812 11:23:42.753316 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.753322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.753407 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.753545 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:42.755439 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.756190 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.757519 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.757947 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.758094 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.763054 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:42.763059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:42.763062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:42.763098 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:42.766877 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.767558 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.767566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.767569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.767708 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:42.768123 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:42.768128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:42.768131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:42.768422 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:42.768740 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.769010 (+ 270us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.769256 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260812 11:23:42.776100 15708 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:23:42.785739 15708 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:42.813195 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.802508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.12.110.193:46809)
0812 11:23:42.804071 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.804083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.804120 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:23:42.804235 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.804244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.804522 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.804742 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:42.805878 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.807333 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.811049 (+ 3716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.811990 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.812181 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.812377 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.812686 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.812849 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1391,"thread_start_us":2216,"threads_started":1}
I20260812 11:23:42.813400 16532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.801894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46809 (local address 127.0.0.1:48926)
0812 11:23:42.803853 (+ 1959us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.803874 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.803995 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.805019 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.805026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.805067 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:42.805686 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.805709 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.807473 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.807481 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.810850 (+ 3369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.810863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.812197 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.812203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.812429 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.812480 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.812620 (+ 140us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.812824 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1781,"mutex_wait_us":235,"spinlock_wait_cycles":1920,"thread_start_us":1620,"threads_started":1}
I20260812 11:23:42.813838 16531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.801566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:37455 (local address 127.0.0.1:57754)
0812 11:23:42.802984 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.803047 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.803419 (+ 372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.805029 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.805032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.805073 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:42.805727 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.805740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.807597 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.807608 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.810508 (+ 2900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.810561 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.812741 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.812747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.812897 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.812924 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.813267 (+ 343us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.813439 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":398,"spinlock_wait_cycles":2304,"thread_start_us":767,"threads_started":1}
I20260812 11:23:42.814360 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.804512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.12.110.195:41709)
0812 11:23:42.805650 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.805658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.805689 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:23:42.805786 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.805795 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.806020 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.806940 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:42.807230 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.808406 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.810914 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.812275 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.813570 (+ 1295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.813735 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.813830 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.814023 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961,"thread_start_us":745,"threads_started":1}
I20260812 11:23:42.814634 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.801851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.12.110.194:37455)
0812 11:23:42.804496 (+ 2645us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.804504 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.804529 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:23:42.804637 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.804644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.804853 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.805076 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:42.805913 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.807441 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.810709 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.812208 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.812398 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.812571 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.814082 (+ 1511us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.814284 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2479,"thread_start_us":2330,"threads_started":1}
I20260812 11:23:42.814846 16536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.803903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41709 (local address 127.0.0.1:48954)
0812 11:23:42.804764 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.804782 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.804884 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.806262 (+ 1378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.806267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.806292 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:42.806708 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.806721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.808553 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.808561 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.810722 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.810738 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.812511 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.812518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.812735 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.812767 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.812831 (+ 64us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.813196 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":4,"thread_start_us":1031,"threads_started":1}
I20260812 11:23:42.818449 16164 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf062fa059e4111bc25df0f412eb413 (DEFAULT_TABLE table=test-table [id=5c744e455e6142fcaae63d8a39524d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:42.818490 15932 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf062fa059e4111bc25df0f412eb413 (DEFAULT_TABLE table=test-table [id=5c744e455e6142fcaae63d8a39524d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:42.819868 16164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf062fa059e4111bc25df0f412eb413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:42.819456 16395 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf062fa059e4111bc25df0f412eb413 (DEFAULT_TABLE table=test-table [id=5c744e455e6142fcaae63d8a39524d2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:42.820009 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf062fa059e4111bc25df0f412eb413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:42.820850 16395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf062fa059e4111bc25df0f412eb413. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:42.837363 16546 tablet_bootstrap.cc:492] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Bootstrap starting.
I20260812 11:23:42.838547 16547 tablet_bootstrap.cc:492] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Bootstrap starting.
I20260812 11:23:42.840173 16541 tablet_bootstrap.cc:492] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Bootstrap starting.
I20260812 11:23:42.842377 16547 tablet_bootstrap.cc:654] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:42.842904 16541 tablet_bootstrap.cc:654] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:42.843008 16546 tablet_bootstrap.cc:654] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:42.843811 16547 log.cc:824] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:42.843885 16541 log.cc:824] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:42.844241 16546 log.cc:824] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:42.848546 16546 tablet_bootstrap.cc:492] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: No bootstrap required, opened a new log
I20260812 11:23:42.848565 16547 tablet_bootstrap.cc:492] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: No bootstrap required, opened a new log
I20260812 11:23:42.848891 16547 ts_tablet_manager.cc:1397] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260812 11:23:42.848948 16546 ts_tablet_manager.cc:1397] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260812 11:23:42.849550 16541 tablet_bootstrap.cc:492] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: No bootstrap required, opened a new log
I20260812 11:23:42.849825 16541 ts_tablet_manager.cc:1397] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260812 11:23:42.858467 16541 raft_consensus.cc:348] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.858954 16541 raft_consensus.cc:374] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:42.859354 16541 raft_consensus.cc:729] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Initialized, Role: FOLLOWER
I20260812 11:23:42.860774 16541 consensus_queue.cc:260] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.861074 16547 raft_consensus.cc:348] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.861065 16546 raft_consensus.cc:348] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.861498 16546 raft_consensus.cc:374] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:42.861599 16547 raft_consensus.cc:374] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:42.861644 16546 raft_consensus.cc:729] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804fa5988396483ea6e6f1d786c7c34a, State: Initialized, Role: FOLLOWER
I20260812 11:23:42.861820 16547 raft_consensus.cc:729] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e0647c24294aa68b67596506bb05fd, State: Initialized, Role: FOLLOWER
I20260812 11:23:42.862381 16546 consensus_queue.cc:260] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.862488 16547 consensus_queue.cc:260] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.862977 16506 heartbeater.cc:503] Master 127.12.110.254:32865 was elected leader, sending a full tablet report...
I20260812 11:23:42.864589 16546 ts_tablet_manager.cc:1428] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260812 11:23:42.865588 16278 heartbeater.cc:503] Master 127.12.110.254:32865 was elected leader, sending a full tablet report...
I20260812 11:23:42.866097 16541 ts_tablet_manager.cc:1428] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
I20260812 11:23:42.867266 16547 ts_tablet_manager.cc:1428] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
W20260812 11:23:42.935631 16508 tablet.cc:2369] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:23:42.967154 16040 tablet.cc:2369] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:23:42.971990 16554 raft_consensus.cc:482] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:42.972290 16554 raft_consensus.cc:504] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.974072 16554 leader_election.cc:288] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455), 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809)
I20260812 11:23:42.977538 16556 raft_consensus.cc:482] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:42.978040 16556 raft_consensus.cc:504] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.980759 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.974473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:58465 (local address 127.12.110.194:37455)
0812 11:23:42.974713 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.974720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.974902 (+ 182us) server_negotiation.cc:407] Connection header received
0812 11:23:42.975185 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.975193 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.975295 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.975407 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:42.975931 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.976813 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.978982 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.979779 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.979962 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.980036 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.980343 (+ 307us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.980472 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:23:42.981385 16320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.974424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:37455 (local address 127.12.110.195:58465)
0812 11:23:42.974808 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.974823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.974984 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.975508 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.975513 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.975595 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:42.975783 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.975791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.977020 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.977028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.978821 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.978835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.979954 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.979960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.980173 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.980202 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.980316 (+ 114us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.980464 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":1,"spinlock_wait_cycles":25472}
I20260812 11:23:42.980993 16556 leader_election.cc:288] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455), 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709)
I20260812 11:23:42.982709 16195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf062fa059e4111bc25df0f412eb413" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd" is_pre_election: true
I20260812 11:23:42.983404 16195 raft_consensus.cc:2454] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 0.
I20260812 11:23:42.984211 16310 leader_election.cc:302] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, f6e0647c24294aa68b67596506bb05fd; no voters:
I20260812 11:23:42.984776 16554 raft_consensus.cc:2781] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:42.984948 16554 raft_consensus.cc:482] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:42.985052 16554 raft_consensus.cc:3037] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:42.985518 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.974622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46809 (local address 127.12.110.195:53745)
0812 11:23:42.975405 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.975425 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.975542 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.975924 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.975928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.975972 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:42.976517 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.976548 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.978160 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.978171 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.980766 (+ 2595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.980795 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.984477 (+ 3682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.984484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.984695 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.984731 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.984815 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.984994 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":5,"thread_start_us":479,"threads_started":1}
I20260812 11:23:42.986425 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.974598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:53745 (local address 127.12.110.193:46809)
0812 11:23:42.974843 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.974851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.975531 (+ 680us) server_negotiation.cc:407] Connection header received
0812 11:23:42.975669 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.975673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.975780 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.975886 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:42.976716 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.977868 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.981123 (+ 3255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.984179 (+ 3056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.985837 (+ 1658us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.985919 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.985973 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.986164 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 11:23:42.987459 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf062fa059e4111bc25df0f412eb413" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a" is_pre_election: true
I20260812 11:23:42.988157 15967 raft_consensus.cc:2454] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 0.
I20260812 11:23:42.989670 16554 raft_consensus.cc:504] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.990389 16554 leader_election.cc:288] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 election: Requested vote from peers f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455), 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809)
I20260812 11:23:42.990847 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf062fa059e4111bc25df0f412eb413" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a"
I20260812 11:23:42.990881 16195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf062fa059e4111bc25df0f412eb413" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd"
I20260812 11:23:42.991081 15967 raft_consensus.cc:3037] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:42.991104 16195 raft_consensus.cc:3037] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:42.993415 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.981435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:35599 (local address 127.12.110.195:41709)
0812 11:23:42.981745 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.981755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.984492 (+ 2737us) server_negotiation.cc:407] Connection header received
0812 11:23:42.984907 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.984918 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.985053 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.985225 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:42.986700 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.988190 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.990953 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.992219 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.992375 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.992448 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.993040 (+ 592us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.993190 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260812 11:23:42.994175 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.982562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41709 (local address 127.12.110.193:35599)
0812 11:23:42.984349 (+ 1787us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.984372 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.984669 (+ 297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.985652 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.985660 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.985693 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:42.986392 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.986419 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.988395 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.988401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.990771 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.990786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.992628 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.992634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.992837 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.992872 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.993624 (+ 752us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.993794 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":1262,"threads_started":1}
I20260812 11:23:42.994772 15967 raft_consensus.cc:2454] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 1.
I20260812 11:23:42.995247 16306 leader_election.cc:302] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, 804fa5988396483ea6e6f1d786c7c34a; no voters:
I20260812 11:23:42.995606 16195 raft_consensus.cc:2454] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 1.
I20260812 11:23:42.995708 16554 raft_consensus.cc:2781] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:42.996099 16433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf062fa059e4111bc25df0f412eb413" candidate_uuid: "804fa5988396483ea6e6f1d786c7c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ea12fe16814693bc4588364d7e7b8f" is_pre_election: true
I20260812 11:23:42.997143 16554 raft_consensus.cc:686] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 1 LEADER]: Becoming Leader. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Running, Role: LEADER
I20260812 11:23:42.997256 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.982644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:37455 (local address 127.12.110.193:44379)
0812 11:23:42.987740 (+ 5096us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:42.987762 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:42.987986 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:42.988579 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:42.988586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:42.988621 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:42.989089 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.989104 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.991055 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.991060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.993790 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:42.993823 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.996328 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.996333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:42.996549 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.996585 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:23:42.996689 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 11:23:42.996849 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4812,"thread_start_us":4555,"threads_started":1}
I20260812 11:23:42.997355 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:42.981402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:44379 (local address 127.12.110.194:37455)
0812 11:23:42.982226 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:42.982234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:42.987860 (+ 5626us) server_negotiation.cc:407] Connection header received
0812 11:23:42.988217 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:42.988222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:42.988305 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:42.988374 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:42.989360 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.990174 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.994125 (+ 3951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:42.995702 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:42.995933 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:42.996152 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:42.996745 (+ 593us) server_negotiation.cc:299] Negotiation successful
0812 11:23:42.996960 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":513,"threads_started":1}
I20260812 11:23:42.997969 16554 consensus_queue.cc:237] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:42.998073 16195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf062fa059e4111bc25df0f412eb413" candidate_uuid: "804fa5988396483ea6e6f1d786c7c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd" is_pre_election: true
I20260812 11:23:42.998486 16195 raft_consensus.cc:2379] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 804fa5988396483ea6e6f1d786c7c34a in current term 1: Already voted for candidate 74ea12fe16814693bc4588364d7e7b8f in this term.
I20260812 11:23:43.003494 15853 leader_election.cc:302] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 804fa5988396483ea6e6f1d786c7c34a; no voters: 74ea12fe16814693bc4588364d7e7b8f, f6e0647c24294aa68b67596506bb05fd
I20260812 11:23:43.004396 16556 raft_consensus.cc:2726] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 11:23:43.005385 15709 catalog_manager.cc:5495] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195). New cstate: current_term: 1 leader_uuid: "74ea12fe16814693bc4588364d7e7b8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } health_report { overall_health: HEALTHY } } }
W20260812 11:23:43.067795 16280 tablet.cc:2369] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 12 11:23:43 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533820, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:43.096302 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.077803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32865 (local address 127.0.0.1:36718)
0812 11:23:43.078954 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:43.079012 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:43.079244 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:43.079728 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:43.079732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:43.080715 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:43.081035 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.081046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.082592 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.082601 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.083894 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.083909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.084883 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.084890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.085041 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.085782 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:43.085801 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:43.088663 (+ 2862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:43.092782 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:43.092803 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:43.092811 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:43.093440 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:43.093922 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:43.093926 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:43.093929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:43.094244 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:43.095163 (+ 919us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:43.095172 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:43.095396 (+ 224us) client_negotiation.cc:769] Sending connection context
0812 11:23:43.095567 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 11:23:43.095880 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":827,"thread_start_us":497,"threads_started":1}
I20260812 11:23:43.096417 16317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.077796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.12.110.254:32865)
0812 11:23:43.078006 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.078012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.079155 (+ 1143us) server_negotiation.cc:407] Connection header received
0812 11:23:43.079411 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.079416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.079490 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.079611 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:43.081247 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.082299 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.084103 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.084632 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.084756 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.088928 (+ 4172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:43.088933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:43.088937 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:43.088964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:43.092543 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:43.093609 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:43.093614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:43.093617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:43.093766 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:43.094459 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:43.094465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:43.094468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:43.094953 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:43.095104 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.095696 (+ 592us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.096071 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":81}
I20260812 11:23:43.102962 15709 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:36718:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:23:43.104084 15709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:43.113066 16164 tablet_service.cc:1467] Processing CreateTablet for tablet 0cbb8a83578a46a68e8945026ebb85b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=638b489c790044a9b283ea983e8a39fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:23:43.113219 16395 tablet_service.cc:1467] Processing CreateTablet for tablet 0cbb8a83578a46a68e8945026ebb85b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=638b489c790044a9b283ea983e8a39fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:23:43.113003 15932 tablet_service.cc:1467] Processing CreateTablet for tablet 0cbb8a83578a46a68e8945026ebb85b8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=638b489c790044a9b283ea983e8a39fc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:23:43.114086 16164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbb8a83578a46a68e8945026ebb85b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:43.114095 16395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbb8a83578a46a68e8945026ebb85b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:43.114095 15932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cbb8a83578a46a68e8945026ebb85b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:43.120678 16547 tablet_bootstrap.cc:492] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: Bootstrap starting.
I20260812 11:23:43.122071 16546 tablet_bootstrap.cc:492] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: Bootstrap starting.
I20260812 11:23:43.122318 16547 tablet_bootstrap.cc:654] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:43.122865 16541 tablet_bootstrap.cc:492] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: Bootstrap starting.
I20260812 11:23:43.124035 16546 tablet_bootstrap.cc:654] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:43.124719 16541 tablet_bootstrap.cc:654] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:43.125242 16547 tablet_bootstrap.cc:492] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: No bootstrap required, opened a new log
I20260812 11:23:43.125414 16547 ts_tablet_manager.cc:1397] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:23:43.126470 16547 raft_consensus.cc:348] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.126766 16546 tablet_bootstrap.cc:492] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: No bootstrap required, opened a new log
I20260812 11:23:43.126935 16547 raft_consensus.cc:374] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:43.127077 16546 ts_tablet_manager.cc:1397] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:23:43.127108 16547 raft_consensus.cc:729] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e0647c24294aa68b67596506bb05fd, State: Initialized, Role: FOLLOWER
I20260812 11:23:43.127578 16547 consensus_queue.cc:260] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.127889 16546 raft_consensus.cc:348] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.128734 16546 raft_consensus.cc:374] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:43.128891 16546 raft_consensus.cc:729] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804fa5988396483ea6e6f1d786c7c34a, State: Initialized, Role: FOLLOWER
I20260812 11:23:43.129333 16546 consensus_queue.cc:260] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.129961 16556 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.130473 16556 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:43.130612 16541 tablet_bootstrap.cc:492] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: No bootstrap required, opened a new log
I20260812 11:23:43.130801 16541 ts_tablet_manager.cc:1397] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260812 11:23:43.131178 16546 ts_tablet_manager.cc:1428] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260812 11:23:43.128345 16557 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.131604 16547 ts_tablet_manager.cc:1428] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260812 11:23:43.131743 16557 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:43.131701 16541 raft_consensus.cc:348] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.131963 16541 raft_consensus.cc:374] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:43.132069 16541 raft_consensus.cc:729] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Initialized, Role: FOLLOWER
I20260812 11:23:43.132325 16541 consensus_queue.cc:260] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.132761 16554 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.133045 16554 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:43.134058 16541 ts_tablet_manager.cc:1428] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:23:43.175115 16557 raft_consensus.cc:482] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:43.175432 16557 raft_consensus.cc:504] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.177215 16557 leader_election.cc:288] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809), 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709)
I20260812 11:23:43.189412 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.178758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:40699 (local address 127.12.110.195:41709)
0812 11:23:43.179107 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.179116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.179694 (+ 578us) server_negotiation.cc:407] Connection header received
0812 11:23:43.179960 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.179969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.180070 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.180185 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:43.180983 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.182063 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.185765 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.187424 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.187650 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.187739 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.188779 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.189003 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:23:43.189412 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.177870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:42837 (local address 127.12.110.193:46809)
0812 11:23:43.178169 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.178178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.178676 (+ 498us) server_negotiation.cc:407] Connection header received
0812 11:23:43.179379 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.179384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.179464 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.179588 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:23:43.180772 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.182000 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.185798 (+ 3798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.187424 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.187672 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.187746 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.188773 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.189000 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260812 11:23:43.189582 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.178477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41709 (local address 127.12.110.194:40699)
0812 11:23:43.179553 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:43.179572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:43.179777 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:43.180205 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:43.180210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:43.180271 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:43.180807 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.180822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.182283 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.182292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.185500 (+ 3208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.185510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.187970 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.187977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.188317 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.188448 (+ 131us) client_negotiation.cc:769] Sending connection context
0812 11:23:43.188584 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 11:23:43.188821 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":318,"thread_start_us":716,"threads_started":1}
I20260812 11:23:43.189610 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.177531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46809 (local address 127.12.110.194:42837)
0812 11:23:43.178450 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:43.178497 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:43.178866 (+ 369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:43.179609 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:43.179614 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:43.179642 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:23:43.180511 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.180533 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.182292 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.182296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.185466 (+ 3170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.185500 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.187944 (+ 2444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.187953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.188281 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.188448 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 11:23:43.188584 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 11:23:43.188801 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":262,"thread_start_us":435,"threads_started":1}
I20260812 11:23:43.190418 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cbb8a83578a46a68e8945026ebb85b8" candidate_uuid: "f6e0647c24294aa68b67596506bb05fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a" is_pre_election: true
I20260812 11:23:43.190824 15967 raft_consensus.cc:2454] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6e0647c24294aa68b67596506bb05fd in term 0.
I20260812 11:23:43.191177 16433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cbb8a83578a46a68e8945026ebb85b8" candidate_uuid: "f6e0647c24294aa68b67596506bb05fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ea12fe16814693bc4588364d7e7b8f" is_pre_election: true
I20260812 11:23:43.191545 16433 raft_consensus.cc:2454] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6e0647c24294aa68b67596506bb05fd in term 0.
I20260812 11:23:43.191591 16068 leader_election.cc:302] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 804fa5988396483ea6e6f1d786c7c34a, f6e0647c24294aa68b67596506bb05fd; no voters:
I20260812 11:23:43.192076 16557 raft_consensus.cc:2781] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:43.192229 16557 raft_consensus.cc:482] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:43.192339 16557 raft_consensus.cc:3037] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:43.194557 16557 raft_consensus.cc:504] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.195209 16557 leader_election.cc:288] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [CANDIDATE]: Term 1 election: Requested vote from peers 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809), 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709)
I20260812 11:23:43.196050 16433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cbb8a83578a46a68e8945026ebb85b8" candidate_uuid: "f6e0647c24294aa68b67596506bb05fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ea12fe16814693bc4588364d7e7b8f"
I20260812 11:23:43.196327 16433 raft_consensus.cc:3037] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:43.196449 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cbb8a83578a46a68e8945026ebb85b8" candidate_uuid: "f6e0647c24294aa68b67596506bb05fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a"
I20260812 11:23:43.196764 15967 raft_consensus.cc:3037] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:43.199379 16433 raft_consensus.cc:2454] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6e0647c24294aa68b67596506bb05fd in term 1.
I20260812 11:23:43.199736 15967 raft_consensus.cc:2454] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6e0647c24294aa68b67596506bb05fd in term 1.
I20260812 11:23:43.200174 16068 leader_election.cc:302] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, f6e0647c24294aa68b67596506bb05fd; no voters:
I20260812 11:23:43.200556 16557 raft_consensus.cc:2781] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:43.201128 16557 raft_consensus.cc:686] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [term 1 LEADER]: Becoming Leader. State: Replica: f6e0647c24294aa68b67596506bb05fd, State: Running, Role: LEADER
I20260812 11:23:43.201695 16557 consensus_queue.cc:237] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.207015 16585 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: TxnStatusTablet state changed. Reason: New leader f6e0647c24294aa68b67596506bb05fd. Latest consensus state: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.207614 16585 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:23:43.208906 16588 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:23:43.209288 16588 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:23:43.209380 15708 catalog_manager.cc:5495] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd reported cstate change: term changed from 0 to 1, leader changed from <none> to f6e0647c24294aa68b67596506bb05fd (127.12.110.194). New cstate: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } health_report { overall_health: UNKNOWN } } }
I20260812 11:23:43.235391 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.227005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.12.110.194:37455)
0812 11:23:43.227432 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.227440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.227703 (+ 263us) server_negotiation.cc:407] Connection header received
0812 11:23:43.228010 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.228019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.228208 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.228400 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:43.229038 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.230518 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.232471 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.233141 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.233331 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.233921 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.234503 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.234725 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.234971 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.235068 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260812 11:23:43.235466 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.227090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:37455 (local address 127.0.0.1:57764)
0812 11:23:43.227570 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:43.227590 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:43.227765 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:43.228490 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:43.228495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:43.228508 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:43.228699 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.228706 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.230765 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.230770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.232207 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.232223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.233355 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.233362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.233546 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.233612 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.234680 (+ 1068us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.234705 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:23:43.234963 (+ 258us) client_negotiation.cc:240] Negotiation successful
0812 11:23:43.235153 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295}
I20260812 11:23:43.244760 15967 raft_consensus.cc:1261] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Refusing update from remote peer f6e0647c24294aa68b67596506bb05fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:43.244800 16433 raft_consensus.cc:1261] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Refusing update from remote peer f6e0647c24294aa68b67596506bb05fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:43.245786 16585 consensus_queue.cc:1035] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:43.246284 16557 consensus_queue.cc:1035] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:43.258859 16556 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: TxnStatusTablet state changed. Reason: New leader f6e0647c24294aa68b67596506bb05fd. Latest consensus state: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.259220 16556 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:43.262085 16554 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: TxnStatusTablet state changed. Reason: New leader f6e0647c24294aa68b67596506bb05fd. Latest consensus state: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.262518 16554 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:43.263758 16592 mvcc.cc:205] Tried to move back new op lower bound from 7317642539999191040 to 7317642539840901120. Current Snapshot: MvccSnapshot[applied={T|T < 7317642539999191040}]
I20260812 11:23:43.266196 16600 mvcc.cc:205] Tried to move back new op lower bound from 7317642539999191040 to 7317642539840901120. Current Snapshot: MvccSnapshot[applied={T|T < 7317642539999191040}]
I20260812 11:23:43.274076 16556 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.274884 16556 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P 804fa5988396483ea6e6f1d786c7c34a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:43.275883 16554 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.276232 16554 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P 74ea12fe16814693bc4588364d7e7b8f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:23:43.283658 16585 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.283743 16557 tablet_replica.cc:439] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6e0647c24294aa68b67596506bb05fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } } }
I20260812 11:23:43.284309 16585 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:23:43.284364 16557 tablet_replica.cc:442] T 0cbb8a83578a46a68e8945026ebb85b8 P f6e0647c24294aa68b67596506bb05fd: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:23:43.300987 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.293236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48964 (local address 127.12.110.195:41709)
0812 11:23:43.293488 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.293496 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.294006 (+ 510us) server_negotiation.cc:407] Connection header received
0812 11:23:43.294256 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.294263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.294384 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.294608 (+ 224us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:43.295177 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.296405 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.297974 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.298778 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.298923 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.299568 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.300267 (+ 699us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.300432 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.300573 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.300663 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 11:23:43.320679 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.312783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:37455 (local address 127.12.110.195:55847)
0812 11:23:43.313494 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:43.313512 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:43.313627 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:43.314146 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:43.314153 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:43.314176 (+ 23us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:43.314832 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.314848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.316684 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.316690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.317942 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.317960 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.319307 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.319316 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.319532 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.319624 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.320131 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.320141 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:23:43.320216 (+ 75us) client_negotiation.cc:240] Negotiation successful
0812 11:23:43.320352 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":369,"threads_started":1}
I20260812 11:23:43.320675 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.312806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:55847 (local address 127.12.110.194:37455)
0812 11:23:43.313063 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.313070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.313598 (+ 528us) server_negotiation.cc:407] Connection header received
0812 11:23:43.313764 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.313770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.313875 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.313986 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:43.315268 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.316465 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.318145 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.319025 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.319197 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.319867 (+ 670us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.320016 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.320111 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.320266 (+ 155us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.320355 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:23:43.333748 15967 raft_consensus.cc:1261] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Refusing update from remote peer 74ea12fe16814693bc4588364d7e7b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:43.333964 16195 raft_consensus.cc:1261] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Refusing update from remote peer 74ea12fe16814693bc4588364d7e7b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:43.334829 16563 consensus_queue.cc:1035] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:43.335207 16554 consensus_queue.cc:1035] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:43.342919 16595 mvcc.cc:205] Tried to move back new op lower bound from 7317642540367728640 to 7317642539004887040. Current Snapshot: MvccSnapshot[applied={T|T < 7317642540367728640}]
I20260812 11:23:43.381095 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.373624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:50459 (local address 127.12.110.195:41709)
0812 11:23:43.374073 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.374080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.375371 (+ 1291us) server_negotiation.cc:407] Connection header received
0812 11:23:43.375567 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.375570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.375636 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.375718 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:43.376421 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.377217 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.379178 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.379610 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.379759 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.380240 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.380373 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.380491 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.380677 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.380761 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260812 11:23:43.381119 16624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.373701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41709 (local address 127.12.110.194:50459)
0812 11:23:43.375199 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:43.375222 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:43.375427 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:43.375760 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:43.375767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:43.375785 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:43.376180 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.376194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.377376 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.377381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.379029 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.379041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.379804 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.379813 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.379952 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.380015 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.380492 (+ 477us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.380506 (+ 14us) client_negotiation.cc:769] Sending connection context
0812 11:23:43.380624 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 11:23:43.380765 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"thread_start_us":1097,"threads_started":1}
I20260812 11:23:43.428292 15709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:43.428956 15709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:43.434124 15709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Sending DeleteTablet for 3 replicas of tablet 6cf062fa059e4111bc25df0f412eb413
I20260812 11:23:43.435564 16164 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf062fa059e4111bc25df0f412eb413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:43 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:57754
I20260812 11:23:43.435619 15932 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf062fa059e4111bc25df0f412eb413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:43 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48926
I20260812 11:23:43.437882 16395 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf062fa059e4111bc25df0f412eb413 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:43 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48954
I20260812 11:23:43.441433 15709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:23:43.442399 15709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:43.452539 15931 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa73366c2e49fca1cf1b299a127029 (DEFAULT_TABLE table=test-table [id=9635118348ee4bef89743dac5729c135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:43.452868 16393 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa73366c2e49fca1cf1b299a127029 (DEFAULT_TABLE table=test-table [id=9635118348ee4bef89743dac5729c135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:43.453276 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa73366c2e49fca1cf1b299a127029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:43.453666 16393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa73366c2e49fca1cf1b299a127029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:43.455061 16631 tablet_replica.cc:330] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: stopping tablet replica
I20260812 11:23:43.455545 16631 raft_consensus.cc:2229] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:43.455947 16631 raft_consensus.cc:2258] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:43.456275 16633 tablet_replica.cc:330] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: stopping tablet replica
I20260812 11:23:43.456686 16633 raft_consensus.cc:2229] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:23:43.456871 16162 tablet_service.cc:1467] Processing CreateTablet for tablet a0fa73366c2e49fca1cf1b299a127029 (DEFAULT_TABLE table=test-table [id=9635118348ee4bef89743dac5729c135]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:43.457386 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0fa73366c2e49fca1cf1b299a127029. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:43.460017 16630 tablet_replica.cc:330] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: stopping tablet replica
I20260812 11:23:43.460492 16546 tablet_bootstrap.cc:492] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: Bootstrap starting.
I20260812 11:23:43.461545 16633 raft_consensus.cc:2258] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:43.462158 16546 tablet_bootstrap.cc:654] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:43.463368 16541 tablet_bootstrap.cc:492] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: Bootstrap starting.
I20260812 11:23:43.464737 16547 tablet_bootstrap.cc:492] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: Bootstrap starting.
I20260812 11:23:43.465217 16541 tablet_bootstrap.cc:654] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:43.466732 16547 tablet_bootstrap.cc:654] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:43.475040 16631 ts_tablet_manager.cc:1905] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:43.479592 16633 ts_tablet_manager.cc:1905] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:43.481986 16546 tablet_bootstrap.cc:492] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: No bootstrap required, opened a new log
I20260812 11:23:43.482291 16546 ts_tablet_manager.cc:1397] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20260812 11:23:43.483222 16546 raft_consensus.cc:348] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.483639 16546 raft_consensus.cc:374] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:43.483776 16546 raft_consensus.cc:729] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804fa5988396483ea6e6f1d786c7c34a, State: Initialized, Role: FOLLOWER
I20260812 11:23:43.484048 16630 raft_consensus.cc:2229] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:43.484163 16546 consensus_queue.cc:260] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.484514 16547 tablet_bootstrap.cc:492] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: No bootstrap required, opened a new log
I20260812 11:23:43.484709 16547 ts_tablet_manager.cc:1397] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.007s
I20260812 11:23:43.485502 16547 raft_consensus.cc:348] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.485764 16547 raft_consensus.cc:374] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:43.485838 16547 raft_consensus.cc:729] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e0647c24294aa68b67596506bb05fd, State: Initialized, Role: FOLLOWER
I20260812 11:23:43.486116 16547 consensus_queue.cc:260] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.486797 16547 ts_tablet_manager.cc:1428] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260812 11:23:43.484478 16630 raft_consensus.cc:2258] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:43.491638 16546 ts_tablet_manager.cc:1428] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260812 11:23:43.496008 16541 tablet_bootstrap.cc:492] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: No bootstrap required, opened a new log
I20260812 11:23:43.496317 16541 ts_tablet_manager.cc:1397] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.001s
I20260812 11:23:43.497434 16541 raft_consensus.cc:348] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.497538 16630 ts_tablet_manager.cc:1905] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:43.497807 16541 raft_consensus.cc:374] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:43.497962 16541 raft_consensus.cc:729] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Initialized, Role: FOLLOWER
I20260812 11:23:43.498345 16541 consensus_queue.cc:260] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.501590 16541 ts_tablet_manager.cc:1428] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260812 11:23:43.506274 16631 ts_tablet_manager.cc:1918] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:43.506482 16631 log.cc:1196] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/6cf062fa059e4111bc25df0f412eb413
I20260812 11:23:43.507133 16631 ts_tablet_manager.cc:1939] T 6cf062fa059e4111bc25df0f412eb413 P 804fa5988396483ea6e6f1d786c7c34a: Deleting consensus metadata
I20260812 11:23:43.509230 15679 catalog_manager.cc:4841] TS 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809): tablet 6cf062fa059e4111bc25df0f412eb413 (table test-table [id=5c744e455e6142fcaae63d8a39524d2c]) successfully deleted
I20260812 11:23:43.515048 16630 ts_tablet_manager.cc:1918] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:43.515503 16630 log.cc:1196] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/6cf062fa059e4111bc25df0f412eb413
I20260812 11:23:43.516317 16630 ts_tablet_manager.cc:1939] T 6cf062fa059e4111bc25df0f412eb413 P f6e0647c24294aa68b67596506bb05fd: Deleting consensus metadata
I20260812 11:23:43.518853 15686 catalog_manager.cc:4841] TS f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455): tablet 6cf062fa059e4111bc25df0f412eb413 (table test-table [id=5c744e455e6142fcaae63d8a39524d2c]) successfully deleted
I20260812 11:23:43.523108 16633 ts_tablet_manager.cc:1918] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:43.523466 16633 log.cc:1196] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/6cf062fa059e4111bc25df0f412eb413
I20260812 11:23:43.524098 16633 ts_tablet_manager.cc:1939] T 6cf062fa059e4111bc25df0f412eb413 P 74ea12fe16814693bc4588364d7e7b8f: Deleting consensus metadata
I20260812 11:23:43.526080 15679 catalog_manager.cc:4841] TS 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709): tablet 6cf062fa059e4111bc25df0f412eb413 (table test-table [id=5c744e455e6142fcaae63d8a39524d2c]) successfully deleted
I20260812 11:23:43.845631 16556 raft_consensus.cc:482] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:43.845942 16556 raft_consensus.cc:504] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.847131 16556 leader_election.cc:288] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455), 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709)
I20260812 11:23:43.848019 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa73366c2e49fca1cf1b299a127029" candidate_uuid: "804fa5988396483ea6e6f1d786c7c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd" is_pre_election: true
I20260812 11:23:43.847987 16433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa73366c2e49fca1cf1b299a127029" candidate_uuid: "804fa5988396483ea6e6f1d786c7c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ea12fe16814693bc4588364d7e7b8f" is_pre_election: true
I20260812 11:23:43.848536 16433 raft_consensus.cc:2454] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804fa5988396483ea6e6f1d786c7c34a in term 0.
I20260812 11:23:43.848515 16185 raft_consensus.cc:2454] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 804fa5988396483ea6e6f1d786c7c34a in term 0.
I20260812 11:23:43.849293 15853 leader_election.cc:302] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, 804fa5988396483ea6e6f1d786c7c34a; no voters:
I20260812 11:23:43.849749 16556 raft_consensus.cc:2781] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:43.850047 16556 raft_consensus.cc:482] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:43.850200 16556 raft_consensus.cc:3037] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:43.852535 16556 raft_consensus.cc:504] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.853255 16556 leader_election.cc:288] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [CANDIDATE]: Term 1 election: Requested vote from peers f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455), 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709)
I20260812 11:23:43.854099 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa73366c2e49fca1cf1b299a127029" candidate_uuid: "804fa5988396483ea6e6f1d786c7c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd"
I20260812 11:23:43.854099 16433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0fa73366c2e49fca1cf1b299a127029" candidate_uuid: "804fa5988396483ea6e6f1d786c7c34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ea12fe16814693bc4588364d7e7b8f"
I20260812 11:23:43.854329 16185 raft_consensus.cc:3037] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:43.854353 16433 raft_consensus.cc:3037] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:43.857754 16185 raft_consensus.cc:2454] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804fa5988396483ea6e6f1d786c7c34a in term 1.
I20260812 11:23:43.857740 16433 raft_consensus.cc:2454] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 804fa5988396483ea6e6f1d786c7c34a in term 1.
I20260812 11:23:43.858307 15853 leader_election.cc:302] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, 804fa5988396483ea6e6f1d786c7c34a; no voters:
I20260812 11:23:43.858654 16556 raft_consensus.cc:2781] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:43.859198 16556 raft_consensus.cc:686] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 1 LEADER]: Becoming Leader. State: Replica: 804fa5988396483ea6e6f1d786c7c34a, State: Running, Role: LEADER
I20260812 11:23:43.859656 16556 consensus_queue.cc:237] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:43.863498 15709 catalog_manager.cc:5495] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a reported cstate change: term changed from 0 to 1, leader changed from <none> to 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193). New cstate: current_term: 1 leader_uuid: "804fa5988396483ea6e6f1d786c7c34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } health_report { overall_health: UNKNOWN } } }
I20260812 11:23:43.947438 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.939927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.12.110.193:46809)
0812 11:23:43.940508 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.940515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.940862 (+ 347us) server_negotiation.cc:407] Connection header received
0812 11:23:43.941120 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.941128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.941245 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.941448 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:43.941860 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.943104 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.944465 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.945230 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.945376 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.945803 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.946657 (+ 854us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.946834 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.946978 (+ 144us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.947074 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":300,"threads_started":1}
I20260812 11:23:43.969852 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.962303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:46607 (local address 127.12.110.194:37455)
0812 11:23:43.962913 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:43.962922 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:43.963386 (+ 464us) server_negotiation.cc:407] Connection header received
0812 11:23:43.963571 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:43.963578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:43.963671 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:43.963796 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:43.964294 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.965386 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.967169 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.967858 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.968009 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.968785 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.968946 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.969077 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:43.969328 (+ 251us) server_negotiation.cc:299] Negotiation successful
0812 11:23:43.969421 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":313,"threads_started":1}
I20260812 11:23:43.970024 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:43.962318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:37455 (local address 127.12.110.193:46607)
0812 11:23:43.963264 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:43.963288 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:43.963437 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:43.963835 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:43.963847 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:43.963862 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:43.964140 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.964149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.965603 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.965609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.966906 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:43.966919 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:43.968050 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:43.968071 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:43.968201 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:43.968468 (+ 267us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:43.969134 (+ 666us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:43.969151 (+ 17us) client_negotiation.cc:769] Sending connection context
0812 11:23:43.969267 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:23:43.969448 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":586,"threads_started":1}
I20260812 11:23:43.980681 16185 raft_consensus.cc:1261] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Refusing update from remote peer 804fa5988396483ea6e6f1d786c7c34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:43.980755 16433 raft_consensus.cc:1261] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Refusing update from remote peer 804fa5988396483ea6e6f1d786c7c34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:43.981353 16641 consensus_queue.cc:1035] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:43.981891 16556 consensus_queue.cc:1035] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:44.031303 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:44.014442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:47643 (local address 127.12.110.193:46809)
0812 11:23:44.014933 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:44.014942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:44.025049 (+ 10107us) server_negotiation.cc:407] Connection header received
0812 11:23:44.025272 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:44.025278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:44.025374 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:44.025477 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:44.026004 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:44.026854 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:44.028775 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:44.029219 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:44.029423 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:44.030477 (+ 1054us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:44.030632 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:44.030744 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:44.030900 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 11:23:44.031002 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353}
I20260812 11:23:44.031473 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:44.014141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46809 (local address 127.12.110.194:47643)
0812 11:23:44.024916 (+ 10775us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:44.024936 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:44.025125 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:44.025555 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:44.025562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:44.025578 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:23:44.025781 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:44.025794 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:44.027003 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:44.027011 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:44.028587 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:44.028605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:44.029435 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:44.029442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:44.029838 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:44.029936 (+ 98us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:23:44.030752 (+ 816us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:23:44.030768 (+ 16us) client_negotiation.cc:769] Sending connection context
0812 11:23:44.030997 (+ 229us) client_negotiation.cc:240] Negotiation successful
0812 11:23:44.031142 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10658,"thread_start_us":11099,"threads_started":1}
W20260812 11:23:44.032984 16650 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 11:23:45.667795 15709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:45.668145 15709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:45.674336 15709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Sending DeleteTablet for 3 replicas of tablet a0fa73366c2e49fca1cf1b299a127029
I20260812 11:23:45.675611 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fa73366c2e49fca1cf1b299a127029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:45 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:57754
I20260812 11:23:45.675668 15931 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fa73366c2e49fca1cf1b299a127029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:45 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48926
I20260812 11:23:45.675743 16395 tablet_service.cc:1514] Processing DeleteTablet for tablet a0fa73366c2e49fca1cf1b299a127029 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:45 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48954
I20260812 11:23:45.688640 16711 tablet_replica.cc:330] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: stopping tablet replica
I20260812 11:23:45.689095 16711 raft_consensus.cc:2229] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:45.689436 16711 raft_consensus.cc:2258] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:45.690299 16713 tablet_replica.cc:330] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: stopping tablet replica
I20260812 11:23:45.690636 16713 raft_consensus.cc:2229] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:23:45.691316 16713 raft_consensus.cc:2258] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:45.692674 16713 ts_tablet_manager.cc:1905] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:45.691706 16711 ts_tablet_manager.cc:1905] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:45.702415 16711 ts_tablet_manager.cc:1918] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:23:45.702762 16711 log.cc:1196] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/a0fa73366c2e49fca1cf1b299a127029
I20260812 11:23:45.702603 16713 ts_tablet_manager.cc:1918] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:23:45.702977 16713 log.cc:1196] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/a0fa73366c2e49fca1cf1b299a127029
I20260812 11:23:45.703411 16711 ts_tablet_manager.cc:1939] T a0fa73366c2e49fca1cf1b299a127029 P f6e0647c24294aa68b67596506bb05fd: Deleting consensus metadata
I20260812 11:23:45.703824 16713 ts_tablet_manager.cc:1939] T a0fa73366c2e49fca1cf1b299a127029 P 804fa5988396483ea6e6f1d786c7c34a: Deleting consensus metadata
I20260812 11:23:45.704929 16712 tablet_replica.cc:330] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: stopping tablet replica
I20260812 11:23:45.705225 15686 catalog_manager.cc:4841] TS f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455): tablet a0fa73366c2e49fca1cf1b299a127029 (table test-table [id=9635118348ee4bef89743dac5729c135]) successfully deleted
I20260812 11:23:45.705331 16712 raft_consensus.cc:2229] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:45.705678 16712 raft_consensus.cc:2258] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:45.706311 15679 catalog_manager.cc:4841] TS 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809): tablet a0fa73366c2e49fca1cf1b299a127029 (table test-table [id=9635118348ee4bef89743dac5729c135]) successfully deleted
I20260812 11:23:45.707135 16712 ts_tablet_manager.cc:1905] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:45.714257 16712 ts_tablet_manager.cc:1918] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:23:45.714470 16712 log.cc:1196] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/a0fa73366c2e49fca1cf1b299a127029
I20260812 11:23:45.715031 16712 ts_tablet_manager.cc:1939] T a0fa73366c2e49fca1cf1b299a127029 P 74ea12fe16814693bc4588364d7e7b8f: Deleting consensus metadata
I20260812 11:23:45.716208 15679 catalog_manager.cc:4841] TS 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709): tablet a0fa73366c2e49fca1cf1b299a127029 (table test-table [id=9635118348ee4bef89743dac5729c135]) successfully deleted
Aug 12 11:23:45 dist-test-slave-h6n0 krb5kdc[15632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:23:45 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:45.774991 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:45.753972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36722 (local address 127.12.110.254:32865)
0812 11:23:45.754762 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:45.754768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:45.754797 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:23:45.754883 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:45.754888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:45.754969 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:45.755137 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:45.756724 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:45.757726 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:45.759751 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:45.760505 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:45.760662 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:45.767284 (+ 6622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:45.767293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:45.767297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:45.767359 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:45.771205 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:45.772254 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:45.772264 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:45.772273 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:45.772464 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:45.772987 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:45.772992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:45.772995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:45.773390 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:45.773711 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:45.774108 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 11:23:45.774445 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":418,"threads_started":1}
I20260812 11:23:45.780762 15709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:23:45.782040 15709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:45.791272 15931 tablet_service.cc:1467] Processing CreateTablet for tablet 5c44b4409a82494589f1a7c8e82f7a33 (DEFAULT_TABLE table=test-table [id=bf3557171dc0498ab4eb1087f186a8b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:45.791524 16162 tablet_service.cc:1467] Processing CreateTablet for tablet 5c44b4409a82494589f1a7c8e82f7a33 (DEFAULT_TABLE table=test-table [id=bf3557171dc0498ab4eb1087f186a8b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:45.791553 16395 tablet_service.cc:1467] Processing CreateTablet for tablet 5c44b4409a82494589f1a7c8e82f7a33 (DEFAULT_TABLE table=test-table [id=bf3557171dc0498ab4eb1087f186a8b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:45.792073 16395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c44b4409a82494589f1a7c8e82f7a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:45.792114 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c44b4409a82494589f1a7c8e82f7a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:45.792222 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c44b4409a82494589f1a7c8e82f7a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:45.798728 16727 tablet_bootstrap.cc:492] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: Bootstrap starting.
I20260812 11:23:45.800954 16733 tablet_bootstrap.cc:492] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: Bootstrap starting.
I20260812 11:23:45.800339 16727 tablet_bootstrap.cc:654] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:45.803092 16733 tablet_bootstrap.cc:654] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:45.803520 16728 tablet_bootstrap.cc:492] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: Bootstrap starting.
I20260812 11:23:45.804906 16727 tablet_bootstrap.cc:492] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: No bootstrap required, opened a new log
I20260812 11:23:45.805234 16727 ts_tablet_manager.cc:1397] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260812 11:23:45.805528 16728 tablet_bootstrap.cc:654] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:45.806310 16733 tablet_bootstrap.cc:492] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: No bootstrap required, opened a new log
I20260812 11:23:45.806260 16727 raft_consensus.cc:348] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.806488 16733 ts_tablet_manager.cc:1397] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260812 11:23:45.806608 16727 raft_consensus.cc:374] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:45.806737 16727 raft_consensus.cc:729] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804fa5988396483ea6e6f1d786c7c34a, State: Initialized, Role: FOLLOWER
I20260812 11:23:45.807078 16727 consensus_queue.cc:260] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.807600 16733 raft_consensus.cc:348] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.807957 16727 ts_tablet_manager.cc:1428] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260812 11:23:45.808053 16733 raft_consensus.cc:374] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:45.808303 16733 raft_consensus.cc:729] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Initialized, Role: FOLLOWER
I20260812 11:23:45.808692 16733 consensus_queue.cc:260] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.809777 16733 ts_tablet_manager.cc:1428] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:23:45.810689 16728 tablet_bootstrap.cc:492] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: No bootstrap required, opened a new log
I20260812 11:23:45.810853 16728 ts_tablet_manager.cc:1397] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260812 11:23:45.811700 16728 raft_consensus.cc:348] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.811977 16728 raft_consensus.cc:374] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:45.812086 16728 raft_consensus.cc:729] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e0647c24294aa68b67596506bb05fd, State: Initialized, Role: FOLLOWER
I20260812 11:23:45.812332 16728 consensus_queue.cc:260] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.812990 16728 ts_tablet_manager.cc:1428] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:23:45.845203 16741 raft_consensus.cc:482] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:45.845505 16741 raft_consensus.cc:504] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.846750 16741 leader_election.cc:288] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809), f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455)
I20260812 11:23:45.847347 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c44b4409a82494589f1a7c8e82f7a33" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a" is_pre_election: true
I20260812 11:23:45.847733 15966 raft_consensus.cc:2454] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 0.
I20260812 11:23:45.847575 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c44b4409a82494589f1a7c8e82f7a33" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd" is_pre_election: true
I20260812 11:23:45.847955 16185 raft_consensus.cc:2454] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 0.
I20260812 11:23:45.848304 16306 leader_election.cc:302] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, 804fa5988396483ea6e6f1d786c7c34a; no voters:
I20260812 11:23:45.848641 16741 raft_consensus.cc:2781] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:45.848774 16741 raft_consensus.cc:482] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:45.848852 16741 raft_consensus.cc:3037] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:45.851397 16741 raft_consensus.cc:504] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.852025 16741 leader_election.cc:288] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 election: Requested vote from peers 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809), f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455)
I20260812 11:23:45.852689 15966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c44b4409a82494589f1a7c8e82f7a33" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a"
I20260812 11:23:45.852826 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c44b4409a82494589f1a7c8e82f7a33" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd"
I20260812 11:23:45.853024 15966 raft_consensus.cc:3037] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:45.853089 16185 raft_consensus.cc:3037] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:45.855376 16185 raft_consensus.cc:2454] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 1.
I20260812 11:23:45.856235 16310 leader_election.cc:302] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, f6e0647c24294aa68b67596506bb05fd; no voters:
I20260812 11:23:45.856344 15966 raft_consensus.cc:2454] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 1.
I20260812 11:23:45.856758 16741 raft_consensus.cc:2781] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:45.857453 16741 raft_consensus.cc:686] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 1 LEADER]: Becoming Leader. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Running, Role: LEADER
I20260812 11:23:45.857865 16741 consensus_queue.cc:237] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:45.861824 15709 catalog_manager.cc:5495] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195). New cstate: current_term: 1 leader_uuid: "74ea12fe16814693bc4588364d7e7b8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } health_report { overall_health: HEALTHY } } }
I20260812 11:23:45.900599 16746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:45.893992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48978 (local address 127.12.110.195:41709)
0812 11:23:45.894767 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:45.894772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:45.894800 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:23:45.894881 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:45.894886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:45.894961 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:45.895090 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:23:45.895632 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:45.896542 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:45.898181 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:45.898728 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:45.898842 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:45.899618 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:23:45.899825 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:23:45.899930 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:45.900193 (+ 263us) server_negotiation.cc:299] Negotiation successful
0812 11:23:45.900271 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":423,"threads_started":1}
I20260812 11:23:45.913965 15966 raft_consensus.cc:1261] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Refusing update from remote peer 74ea12fe16814693bc4588364d7e7b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:45.913970 16185 raft_consensus.cc:1261] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Refusing update from remote peer 74ea12fe16814693bc4588364d7e7b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:45.914645 16742 consensus_queue.cc:1035] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:45.915215 16741 consensus_queue.cc:1035] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:45.982653 15709 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:45.982966 15709 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:45.987244 15709 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Sending DeleteTablet for 3 replicas of tablet 5c44b4409a82494589f1a7c8e82f7a33
I20260812 11:23:45.988468 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c44b4409a82494589f1a7c8e82f7a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:45 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:57754
I20260812 11:23:45.988613 15931 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c44b4409a82494589f1a7c8e82f7a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:45 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48926
I20260812 11:23:45.988633 16395 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c44b4409a82494589f1a7c8e82f7a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:45 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48954
I20260812 11:23:45.988884 16711 tablet_replica.cc:330] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: stopping tablet replica
I20260812 11:23:45.989032 16712 tablet_replica.cc:330] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: stopping tablet replica
I20260812 11:23:45.989009 16713 tablet_replica.cc:330] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: stopping tablet replica
I20260812 11:23:45.989176 16711 raft_consensus.cc:2229] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:45.989264 16712 raft_consensus.cc:2229] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:23:45.989279 16713 raft_consensus.cc:2229] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:45.989487 16711 raft_consensus.cc:2258] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:45.989530 16713 raft_consensus.cc:2258] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:45.989662 16712 raft_consensus.cc:2258] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:45.990427 15709 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36722:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260812 11:23:45.990891 16713 ts_tablet_manager.cc:1905] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:45.991065 16712 ts_tablet_manager.cc:1905] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:45.991353 16711 ts_tablet_manager.cc:1905] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260812 11:23:45.991439 15709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:23:45.998308 16713 ts_tablet_manager.cc:1918] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:45.998603 16713 log.cc:1196] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/5c44b4409a82494589f1a7c8e82f7a33
I20260812 11:23:45.998811 16711 ts_tablet_manager.cc:1918] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:45.999158 16711 log.cc:1196] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/5c44b4409a82494589f1a7c8e82f7a33
I20260812 11:23:45.999336 16713 ts_tablet_manager.cc:1939] T 5c44b4409a82494589f1a7c8e82f7a33 P 804fa5988396483ea6e6f1d786c7c34a: Deleting consensus metadata
I20260812 11:23:45.999771 16711 ts_tablet_manager.cc:1939] T 5c44b4409a82494589f1a7c8e82f7a33 P f6e0647c24294aa68b67596506bb05fd: Deleting consensus metadata
I20260812 11:23:46.000036 16712 ts_tablet_manager.cc:1918] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:23:46.000334 16712 log.cc:1196] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/5c44b4409a82494589f1a7c8e82f7a33
I20260812 11:23:46.000874 16712 ts_tablet_manager.cc:1939] T 5c44b4409a82494589f1a7c8e82f7a33 P 74ea12fe16814693bc4588364d7e7b8f: Deleting consensus metadata
I20260812 11:23:46.001425 15686 catalog_manager.cc:4841] TS f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455): tablet 5c44b4409a82494589f1a7c8e82f7a33 (table test-table [id=bf3557171dc0498ab4eb1087f186a8b0]) successfully deleted
I20260812 11:23:46.001705 15679 catalog_manager.cc:4841] TS 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809): tablet 5c44b4409a82494589f1a7c8e82f7a33 (table test-table [id=bf3557171dc0498ab4eb1087f186a8b0]) successfully deleted
I20260812 11:23:46.002244 16395 tablet_service.cc:1467] Processing CreateTablet for tablet 730634c49a62447d86d7a9d222553cbe (DEFAULT_TABLE table=test-table [id=d10d74632c994cd89f601b35296662bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:46.002342 16162 tablet_service.cc:1467] Processing CreateTablet for tablet 730634c49a62447d86d7a9d222553cbe (DEFAULT_TABLE table=test-table [id=d10d74632c994cd89f601b35296662bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:46.002386 15931 tablet_service.cc:1467] Processing CreateTablet for tablet 730634c49a62447d86d7a9d222553cbe (DEFAULT_TABLE table=test-table [id=d10d74632c994cd89f601b35296662bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:23:46.002904 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730634c49a62447d86d7a9d222553cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:46.002928 15931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730634c49a62447d86d7a9d222553cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:46.003034 16395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 730634c49a62447d86d7a9d222553cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:46.004642 15679 catalog_manager.cc:4841] TS 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709): tablet 5c44b4409a82494589f1a7c8e82f7a33 (table test-table [id=bf3557171dc0498ab4eb1087f186a8b0]) successfully deleted
I20260812 11:23:46.009498 16727 tablet_bootstrap.cc:492] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: Bootstrap starting.
I20260812 11:23:46.010607 16733 tablet_bootstrap.cc:492] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: Bootstrap starting.
I20260812 11:23:46.011096 16728 tablet_bootstrap.cc:492] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: Bootstrap starting.
I20260812 11:23:46.011260 16727 tablet_bootstrap.cc:654] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:46.012468 16733 tablet_bootstrap.cc:654] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:46.012863 16728 tablet_bootstrap.cc:654] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:46.013973 16727 tablet_bootstrap.cc:492] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: No bootstrap required, opened a new log
I20260812 11:23:46.014148 16727 ts_tablet_manager.cc:1397] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260812 11:23:46.015116 16727 raft_consensus.cc:348] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.015225 16733 tablet_bootstrap.cc:492] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: No bootstrap required, opened a new log
I20260812 11:23:46.015378 16727 raft_consensus.cc:374] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:46.015436 16733 ts_tablet_manager.cc:1397] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260812 11:23:46.015498 16727 raft_consensus.cc:729] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804fa5988396483ea6e6f1d786c7c34a, State: Initialized, Role: FOLLOWER
I20260812 11:23:46.015697 16728 tablet_bootstrap.cc:492] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: No bootstrap required, opened a new log
I20260812 11:23:46.015889 16728 ts_tablet_manager.cc:1397] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:23:46.015856 16727 consensus_queue.cc:260] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.016433 16727 ts_tablet_manager.cc:1428] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:23:46.016390 16733 raft_consensus.cc:348] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.016705 16733 raft_consensus.cc:374] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:46.016856 16733 raft_consensus.cc:729] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Initialized, Role: FOLLOWER
I20260812 11:23:46.016758 16728 raft_consensus.cc:348] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.017037 16728 raft_consensus.cc:374] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:46.017120 16728 raft_consensus.cc:729] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e0647c24294aa68b67596506bb05fd, State: Initialized, Role: FOLLOWER
I20260812 11:23:46.017154 16733 consensus_queue.cc:260] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.017400 16728 consensus_queue.cc:260] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.018019 16728 ts_tablet_manager.cc:1428] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 11:23:46.019083 16733 ts_tablet_manager.cc:1428] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260812 11:23:46.103646 16751 raft_consensus.cc:482] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:23:46.103982 16751 raft_consensus.cc:504] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.105008 16751 leader_election.cc:288] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809), f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455)
I20260812 11:23:46.105918 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730634c49a62447d86d7a9d222553cbe" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a" is_pre_election: true
I20260812 11:23:46.105918 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730634c49a62447d86d7a9d222553cbe" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd" is_pre_election: true
I20260812 11:23:46.106271 16185 raft_consensus.cc:2454] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 0.
I20260812 11:23:46.106271 15967 raft_consensus.cc:2454] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 0.
I20260812 11:23:46.106884 16306 leader_election.cc:302] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, 804fa5988396483ea6e6f1d786c7c34a; no voters:
I20260812 11:23:46.107342 16751 raft_consensus.cc:2781] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:23:46.107589 16751 raft_consensus.cc:482] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:23:46.107713 16751 raft_consensus.cc:3037] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:46.110316 16751 raft_consensus.cc:504] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.111104 16751 leader_election.cc:288] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 election: Requested vote from peers 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809), f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455)
I20260812 11:23:46.111855 16185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730634c49a62447d86d7a9d222553cbe" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6e0647c24294aa68b67596506bb05fd"
I20260812 11:23:46.111922 15967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "730634c49a62447d86d7a9d222553cbe" candidate_uuid: "74ea12fe16814693bc4588364d7e7b8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804fa5988396483ea6e6f1d786c7c34a"
I20260812 11:23:46.112174 16185 raft_consensus.cc:3037] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:46.112174 15967 raft_consensus.cc:3037] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:46.115592 16185 raft_consensus.cc:2454] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 1.
I20260812 11:23:46.115592 15967 raft_consensus.cc:2454] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 74ea12fe16814693bc4588364d7e7b8f in term 1.
I20260812 11:23:46.116133 16310 leader_election.cc:302] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74ea12fe16814693bc4588364d7e7b8f, f6e0647c24294aa68b67596506bb05fd; no voters:
I20260812 11:23:46.116478 16751 raft_consensus.cc:2781] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:46.116693 16751 raft_consensus.cc:686] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 1 LEADER]: Becoming Leader. State: Replica: 74ea12fe16814693bc4588364d7e7b8f, State: Running, Role: LEADER
I20260812 11:23:46.117110 16751 consensus_queue.cc:237] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } }
I20260812 11:23:46.120606 15709 catalog_manager.cc:5495] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195). New cstate: current_term: 1 leader_uuid: "74ea12fe16814693bc4588364d7e7b8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ea12fe16814693bc4588364d7e7b8f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41709 } health_report { overall_health: HEALTHY } } }
I20260812 11:23:46.170782 16185 raft_consensus.cc:1261] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Refusing update from remote peer 74ea12fe16814693bc4588364d7e7b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:46.170779 15967 raft_consensus.cc:1261] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Refusing update from remote peer 74ea12fe16814693bc4588364d7e7b8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:23:46.171607 16742 consensus_queue.cc:1035] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6e0647c24294aa68b67596506bb05fd" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 37455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:23:46.171993 16751 consensus_queue.cc:1035] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "804fa5988396483ea6e6f1d786c7c34a" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:23:46.201223 16683 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260812 11:23:47.804141 15707 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:47.804360 15707 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36722:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:23:47.808633 15707 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 394a5f95769a44488fc15a94828b61d1: Sending DeleteTablet for 3 replicas of tablet 730634c49a62447d86d7a9d222553cbe
I20260812 11:23:47.809494 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet 730634c49a62447d86d7a9d222553cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:47 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:57754
I20260812 11:23:47.809531 16395 tablet_service.cc:1514] Processing DeleteTablet for tablet 730634c49a62447d86d7a9d222553cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:47 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48954
I20260812 11:23:47.809553 15931 tablet_service.cc:1514] Processing DeleteTablet for tablet 730634c49a62447d86d7a9d222553cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:23:47 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:48926
I20260812 11:23:47.810300 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15825
I20260812 11:23:47.821764 16799 tablet_replica.cc:330] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: stopping tablet replica
I20260812 11:23:47.822142 16799 raft_consensus.cc:2229] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:47.822593 16799 raft_consensus.cc:2258] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:47.824010 16799 ts_tablet_manager.cc:1905] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:47.826947 16798 tablet_replica.cc:330] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: stopping tablet replica
I20260812 11:23:47.827129 16797 tablet_replica.cc:330] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: stopping tablet replica
I20260812 11:23:47.827207 16798 raft_consensus.cc:2229] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:23:47.827453 16797 raft_consensus.cc:2229] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:23:47.827509 16798 raft_consensus.cc:2258] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:47.827971 16797 raft_consensus.cc:2258] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:23:47.829154 16797 ts_tablet_manager.cc:1905] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:47.829715 16798 ts_tablet_manager.cc:1905] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:23:47.832777 16799 ts_tablet_manager.cc:1918] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:23:47.832921 16799 log.cc:1196] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/730634c49a62447d86d7a9d222553cbe
I20260812 11:23:47.833858 16799 ts_tablet_manager.cc:1939] T 730634c49a62447d86d7a9d222553cbe P 804fa5988396483ea6e6f1d786c7c34a: Deleting consensus metadata
I20260812 11:23:47.835091 15679 catalog_manager.cc:4841] TS 804fa5988396483ea6e6f1d786c7c34a (127.12.110.193:46809): tablet 730634c49a62447d86d7a9d222553cbe (table test-table [id=d10d74632c994cd89f601b35296662bd]) successfully deleted
I20260812 11:23:47.835834 16797 ts_tablet_manager.cc:1918] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:23:47.835983 16797 log.cc:1196] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/730634c49a62447d86d7a9d222553cbe
I20260812 11:23:47.836513 16797 ts_tablet_manager.cc:1939] T 730634c49a62447d86d7a9d222553cbe P 74ea12fe16814693bc4588364d7e7b8f: Deleting consensus metadata
I20260812 11:23:47.836928 16798 ts_tablet_manager.cc:1918] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260812 11:23:47.837097 16798 log.cc:1196] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/730634c49a62447d86d7a9d222553cbe
I20260812 11:23:47.837654 16798 ts_tablet_manager.cc:1939] T 730634c49a62447d86d7a9d222553cbe P f6e0647c24294aa68b67596506bb05fd: Deleting consensus metadata
I20260812 11:23:47.837632 15679 catalog_manager.cc:4841] TS 74ea12fe16814693bc4588364d7e7b8f (127.12.110.195:41709): tablet 730634c49a62447d86d7a9d222553cbe (table test-table [id=d10d74632c994cd89f601b35296662bd]) successfully deleted
Aug 12 11:23:47 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:47.838918 15686 catalog_manager.cc:4841] TS f6e0647c24294aa68b67596506bb05fd (127.12.110.194:37455): tablet 730634c49a62447d86d7a9d222553cbe (table test-table [id=d10d74632c994cd89f601b35296662bd]) successfully deleted
I20260812 11:23:47.847335 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:47.813644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.12.110.193:46809)
0812 11:23:47.821642 (+ 7998us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:47.821652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:47.821688 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:23:47.821775 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:47.821785 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:47.821896 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:47.822060 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:47.825364 (+ 3304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:47.827019 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:47.829069 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:47.829721 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:47.829867 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:47.839279 (+ 9412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:47.839296 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:47.839315 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:47.839366 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:47.843176 (+ 3810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:47.844219 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:47.844229 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:47.844232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:47.844417 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:47.845222 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:47.845227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:47.845230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:47.845743 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:47.846114 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:47.846789 (+ 675us) server_negotiation.cc:299] Negotiation successful
0812 11:23:47.847058 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7673,"thread_start_us":7482,"threads_started":1}
W20260812 11:23:47.848245 16028 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48944
W20260812 11:23:47.849128 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15825: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:47.849426 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15825
I20260812 11:23:47.887924 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16043
Aug 12 11:23:47 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:23:47.906975 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:47.888838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57768 (local address 127.12.110.194:37455)
0812 11:23:47.889399 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:47.889403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:47.889420 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:23:47.889477 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:47.889480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:47.889544 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:47.889641 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:47.891108 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:47.891957 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:47.893449 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:47.893851 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:47.893961 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:47.898757 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:47.898770 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:47.898785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:47.898818 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:47.903089 (+ 4271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:47.904205 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:47.904213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:47.904215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:47.904401 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:47.904826 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:47.904829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:47.904832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:47.905240 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:47.905662 (+ 422us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:47.906193 (+ 531us) server_negotiation.cc:299] Negotiation successful
0812 11:23:47.906451 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":296,"threads_started":1}
W20260812 11:23:47.907888 16271 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57768
W20260812 11:23:47.908645 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16043: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:47.908818 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16043
I20260812 11:23:47.943830 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16283
Aug 12 11:23:47 dist-test-slave-h6n0 krb5kdc[15632](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:23:47.963595 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:47.945076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.12.110.195:41709)
0812 11:23:47.945896 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:47.945902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:47.945922 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:23:47.945992 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:47.945997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:47.946083 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:47.946258 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:47.948124 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:47.949106 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:47.950331 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:47.950935 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:47.951037 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:47.956143 (+ 5106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:47.956159 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:47.956178 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:47.956217 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:47.959266 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:47.960705 (+ 1439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:47.960717 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:47.960720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:47.961089 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:47.961728 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:47.961733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:47.961735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:47.962162 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:47.962439 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:47.962942 (+ 503us) server_negotiation.cc:299] Negotiation successful
0812 11:23:47.963269 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":442,"threads_started":1}
W20260812 11:23:47.964664 16495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48990
W20260812 11:23:47.965598 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16283: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:47.965972 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16283
I20260812 11:23:48.005290 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15652
I20260812 11:23:48.007160 15776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:48.138321 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15652
2026-08-12T11:23:48Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9645 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:48 dist-test-slave-h6n0 krb5kdc[16812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:48 dist-test-slave-h6n0 krb5kdc[16812](info): set up 2 sockets
Aug 12 11:23:48 dist-test-slave-h6n0 krb5kdc[16812](info): commencing operation
krb5kdc: starting...
W20260812 11:23:49.561363 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.357s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:23:49 dist-test-slave-h6n0 krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:23:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:23:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:23:49.819731 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37193
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37193
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:50.174073 16839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:50.174666 16839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:50.174896 16839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:50.183106 16839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:23:50.183218 16839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:50.183341 16839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:50.183434 16839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:23:50.183528 16839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:23:50.193845 16839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:50.195187 16839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:50.196509 16839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:50.208210 16847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:50.208551 16850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:50.209023 16852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:50.209870 16839 server_base.cc:1034] running on GCE node
I20260812 11:23:50.210798 16839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:50.212211 16839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:50.213434 16839 hybrid_clock.cc:648] HybridClock initialized: now 1786533830213390 us; error 32 us; skew 500 ppm
I20260812 11:23:50.213820 16839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:50 dist-test-slave-h6n0 krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:50.217304 16839 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:23:50.218894 16839 webserver.cc:460] Webserver started at http://127.12.110.254:40287/ using document root <none> and password file <none>
I20260812 11:23:50.219626 16839 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:50.219799 16839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:50.220139 16839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:50.223242 16839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "0a45fff1b56d4d838d5b6f77e454a6a8"
format_stamp: "Formatted at 2026-08-12 11:23:50 on dist-test-slave-h6n0"
server_key: "a37db9432168e4c2dcc6a73dcef1546e"
server_key_iv: "d027a22e6f59f1333a206534b8c0e8fd"
server_key_version: "encryptionkey@0"
I20260812 11:23:50.224031 16839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "0a45fff1b56d4d838d5b6f77e454a6a8"
format_stamp: "Formatted at 2026-08-12 11:23:50 on dist-test-slave-h6n0"
server_key: "a37db9432168e4c2dcc6a73dcef1546e"
server_key_iv: "d027a22e6f59f1333a206534b8c0e8fd"
server_key_version: "encryptionkey@0"
I20260812 11:23:50.229262 16839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 11:23:50.232807 16863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:50.234086 16839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 11:23:50.234275 16839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "0a45fff1b56d4d838d5b6f77e454a6a8"
format_stamp: "Formatted at 2026-08-12 11:23:50 on dist-test-slave-h6n0"
server_key: "a37db9432168e4c2dcc6a73dcef1546e"
server_key_iv: "d027a22e6f59f1333a206534b8c0e8fd"
server_key_version: "encryptionkey@0"
I20260812 11:23:50.234489 16839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:50.258832 16839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:50.263469 16839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:50.263778 16839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:50.289288 16839 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37193
I20260812 11:23:50.289312 16938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37193 every 8 connection(s)
I20260812 11:23:50.290773 16839 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:23:50.293828 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16839
I20260812 11:23:50.294082 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:23:50.294787 12731 external_mini_cluster.cc:1428] Setting key 895793690b42cee8f6ec8d17e4db7e44
I20260812 11:23:50.294855 16940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:50.309929 16940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8: Bootstrap starting.
Aug 12 11:23:50 dist-test-slave-h6n0 krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:50.313301 16940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:50.315019 16940 log.cc:824] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:50.317983 16940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8: No bootstrap required, opened a new log
I20260812 11:23:50.320973 16944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:50.296736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.12.110.254:37193)
0812 11:23:50.297590 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:50.297601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:50.297645 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:23:50.299911 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:50.299932 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:50.300301 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:50.300629 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:50.302661 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:50.303992 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:50.305734 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:50.306386 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:50.306507 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:50.312230 (+ 5723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:50.312246 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:50.312268 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:50.312355 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:50.316243 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:50.317096 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:50.317103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:50.317110 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:50.317286 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:50.317667 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:50.317672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:50.317677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:50.318266 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:50.318659 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:50.319198 (+ 539us) server_negotiation.cc:299] Negotiation successful
0812 11:23:50.319744 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":383,"threads_started":1}
I20260812 11:23:50.328794 16940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37193 } }
I20260812 11:23:50.329207 16940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:50.329348 16940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a45fff1b56d4d838d5b6f77e454a6a8, State: Initialized, Role: FOLLOWER
I20260812 11:23:50.330011 16940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37193 } }
I20260812 11:23:50.330240 16940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:23:50.330343 16940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:23:50.330463 16940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:50.333173 16940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37193 } }
I20260812 11:23:50.333670 16940 leader_election.cc:302] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a45fff1b56d4d838d5b6f77e454a6a8; no voters:
I20260812 11:23:50.334352 16940 leader_election.cc:288] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:23:50.334864 16946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:50.336022 16946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [term 1 LEADER]: Becoming Leader. State: Replica: 0a45fff1b56d4d838d5b6f77e454a6a8, State: Running, Role: LEADER
I20260812 11:23:50.336792 16946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37193 } }
I20260812 11:23:50.337059 16940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:23:50.342403 16949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a45fff1b56d4d838d5b6f77e454a6a8. Latest consensus state: current_term: 1 leader_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37193 } } }
I20260812 11:23:50.343103 16949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:50.342370 16947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a45fff1b56d4d838d5b6f77e454a6a8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37193 } } }
I20260812 11:23:50.343617 16947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:50.347873 16963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:23:50.352300 16963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:23:50.362214 16963 catalog_manager.cc:1347] Generated new cluster ID: a0b8a73204fc4bb28fc2b091d01f576e
I20260812 11:23:50.362349 16963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:23:50.378166 16963 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:23:50.380111 16963 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:23:50.395836 16963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a45fff1b56d4d838d5b6f77e454a6a8: Generated new TSK 0
I20260812 11:23:50.396762 16963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:23:50.462824 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:50.824868 16980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:50.825212 16980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:50.825387 16980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:50.832800 16980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:50.832968 16980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:50.833142 16980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:23:50.843649 16980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:50.845047 16980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:50.846410 16980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:50.861127 16985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:50.861123 16987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:50.861162 16993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:50.862165 16980 server_base.cc:1034] running on GCE node
I20260812 11:23:50.862884 16980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:50.863822 16980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:50.865046 16980 hybrid_clock.cc:648] HybridClock initialized: now 1786533830864976 us; error 50 us; skew 500 ppm
I20260812 11:23:50.865520 16980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:50 dist-test-slave-h6n0 krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:50.869506 16980 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:23:50.871361 16980 webserver.cc:460] Webserver started at http://127.12.110.193:38715/ using document root <none> and password file <none>
I20260812 11:23:50.872062 16980 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:50.872169 16980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:50.872433 16980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:50.876186 16980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "a27fc6cde33a4736a77cca37f61601b0"
format_stamp: "Formatted at 2026-08-12 11:23:50 on dist-test-slave-h6n0"
server_key: "2a14c053327ce3304c8799de68584efb"
server_key_iv: "bf9900c4f9bab705fcab5ba97276744f"
server_key_version: "encryptionkey@0"
I20260812 11:23:50.877017 16980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "a27fc6cde33a4736a77cca37f61601b0"
format_stamp: "Formatted at 2026-08-12 11:23:50 on dist-test-slave-h6n0"
server_key: "2a14c053327ce3304c8799de68584efb"
server_key_iv: "bf9900c4f9bab705fcab5ba97276744f"
server_key_version: "encryptionkey@0"
I20260812 11:23:50.883651 16980 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260812 11:23:50.887441 17003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:50.889511 16980 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:23:50.889715 16980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "a27fc6cde33a4736a77cca37f61601b0"
format_stamp: "Formatted at 2026-08-12 11:23:50 on dist-test-slave-h6n0"
server_key: "2a14c053327ce3304c8799de68584efb"
server_key_iv: "bf9900c4f9bab705fcab5ba97276744f"
server_key_version: "encryptionkey@0"
I20260812 11:23:50.889984 16980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:50.908955 16980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:50.913697 16980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:50.914048 16980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:50.915293 16980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:50.916754 16980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:50.916865 16980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:50.917029 16980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:50.917112 16980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:50.955850 16980 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:33303
I20260812 11:23:50.955899 17165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:33303 every 8 connection(s)
I20260812 11:23:50.957185 16980 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:23:50.960451 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16980
I20260812 11:23:50.960691 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:23:50.961371 12731 external_mini_cluster.cc:1428] Setting key 003eea791856c91a66adb3f4427264d1
Aug 12 11:23:50 dist-test-slave-h6n0 krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533830, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:50.982230 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:50.960011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:50235 (local address 127.12.110.254:37193)
0812 11:23:50.960575 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:50.960582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:50.961672 (+ 1090us) server_negotiation.cc:407] Connection header received
0812 11:23:50.963941 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:50.963946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:50.964026 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:50.964145 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:50.965716 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:50.966919 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:50.969465 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:50.970154 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:50.970304 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:50.975255 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:50.975263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:50.975267 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:50.975321 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:50.978638 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:50.979676 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:50.979681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:50.979683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:50.979809 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:50.980378 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:50.980382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:50.980384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:50.980673 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:50.980788 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:50.981760 (+ 972us) server_negotiation.cc:299] Negotiation successful
0812 11:23:50.981949 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":270,"threads_started":1}
I20260812 11:23:50.983956 17176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:50.960561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37193 (local address 127.12.110.193:50235)
0812 11:23:50.961495 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:50.961544 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:50.963663 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:50.964409 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:50.964417 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:50.964966 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:50.965530 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:50.965547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:50.967129 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:50.967135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:50.969263 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:50.969296 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:50.970438 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:50.970447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:50.970813 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:50.971561 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:50.971590 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:50.974861 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:50.978802 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:50.978814 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:50.978834 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:50.979531 (+ 697us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:50.979942 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:50.979950 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:50.979956 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:50.980235 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:50.980818 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:50.980829 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:50.981338 (+ 509us) client_negotiation.cc:769] Sending connection context
0812 11:23:50.981677 (+ 339us) client_negotiation.cc:240] Negotiation successful
0812 11:23:50.982316 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":408,"threads_started":1}
I20260812 11:23:50.985848 17170 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37193
I20260812 11:23:50.986203 17170 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:50.987126 17170 heartbeater.cc:511] Master 127.12.110.254:37193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
I20260812 11:23:50.990046 16888 ts_manager.cc:194] Registered new tserver with Master: a27fc6cde33a4736a77cca37f61601b0 (127.12.110.193:33303)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
I20260812 11:23:50.991655 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:50235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:23:51.021793 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:51.376482 17180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:51.376884 17180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:51.377067 17180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:51.384646 17180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:51.384812 17180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:51.384999 17180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:23:51.395751 17180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:51.397332 17180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:51.398773 17180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:51.412600 17194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:51.412672 17188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:51.412693 17186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:51.413843 17180 server_base.cc:1034] running on GCE node
I20260812 11:23:51.414568 17180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:51.415588 17180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:51.416827 17180 hybrid_clock.cc:648] HybridClock initialized: now 1786533831416770 us; error 34 us; skew 500 ppm
I20260812 11:23:51.417179 17180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:51 dist-test-slave-h6n0 krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:51.420480 17180 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:23:51.422039 17180 webserver.cc:460] Webserver started at http://127.12.110.194:37497/ using document root <none> and password file <none>
I20260812 11:23:51.422932 17180 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:51.423086 17180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:51.423431 17180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:51.426451 17180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "e1a4cc7cb3b44516b1a0a75ad05b628d"
format_stamp: "Formatted at 2026-08-12 11:23:51 on dist-test-slave-h6n0"
server_key: "f9fa3b390dd0931c64ca4fbcab9e72a6"
server_key_iv: "7fe8a0843a8927a0afd1192438959912"
server_key_version: "encryptionkey@0"
I20260812 11:23:51.427407 17180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "e1a4cc7cb3b44516b1a0a75ad05b628d"
format_stamp: "Formatted at 2026-08-12 11:23:51 on dist-test-slave-h6n0"
server_key: "f9fa3b390dd0931c64ca4fbcab9e72a6"
server_key_iv: "7fe8a0843a8927a0afd1192438959912"
server_key_version: "encryptionkey@0"
I20260812 11:23:51.433219 17180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:23:51.438140 17207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:51.439881 17180 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:23:51.440076 17180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "e1a4cc7cb3b44516b1a0a75ad05b628d"
format_stamp: "Formatted at 2026-08-12 11:23:51 on dist-test-slave-h6n0"
server_key: "f9fa3b390dd0931c64ca4fbcab9e72a6"
server_key_iv: "7fe8a0843a8927a0afd1192438959912"
server_key_version: "encryptionkey@0"
I20260812 11:23:51.440294 17180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:51.454073 17180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:51.460492 17180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:51.460902 17180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:51.462285 17180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:51.464165 17180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:51.464298 17180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:51.464471 17180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:51.464555 17180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:51.507900 17180 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:41207
I20260812 11:23:51.507953 17413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:41207 every 8 connection(s)
I20260812 11:23:51.509724 17180 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:23:51.510347 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17180
I20260812 11:23:51.510636 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:23:51.511284 12731 external_mini_cluster.cc:1428] Setting key d3d0111327fab9364ee0659681b4588c
Aug 12 11:23:51 dist-test-slave-h6n0 krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:51.536240 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:51.514000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:40947 (local address 127.12.110.254:37193)
0812 11:23:51.514669 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:51.514677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:51.516846 (+ 2169us) server_negotiation.cc:407] Connection header received
0812 11:23:51.519114 (+ 2268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:51.519121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:51.519232 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:51.519404 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:51.521180 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:51.521982 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:51.523895 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:51.524396 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:51.524552 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:51.529290 (+ 4738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:51.529296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:51.529299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:51.529335 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:51.532892 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:51.533720 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:51.533730 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:51.533732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:51.533864 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:51.534337 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:51.534342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:51.534344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:51.534704 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:51.534867 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:51.535613 (+ 746us) server_negotiation.cc:299] Negotiation successful
0812 11:23:51.535861 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":354,"threads_started":1}
I20260812 11:23:51.537333 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:51.515282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37193 (local address 127.12.110.194:40947)
0812 11:23:51.516549 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:51.516626 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:51.518785 (+ 2159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:51.519692 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:51.519702 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:51.520296 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:51.520983 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:51.521002 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:51.522159 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:51.522165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:51.523707 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:51.523721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:51.524661 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:51.524669 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:51.524931 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:51.525863 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:51.525896 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:51.529030 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:51.533070 (+ 4040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:51.533080 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:51.533099 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:51.533581 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:51.533993 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:51.533997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:51.534001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:51.534203 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:51.534847 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:51.534855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:51.535218 (+ 363us) client_negotiation.cc:769] Sending connection context
0812 11:23:51.535501 (+ 283us) client_negotiation.cc:240] Negotiation successful
0812 11:23:51.536079 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":435,"threads_started":1}
I20260812 11:23:51.539297 17415 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37193
I20260812 11:23:51.539819 17415 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
I20260812 11:23:51.540616 17415 heartbeater.cc:511] Master 127.12.110.254:37193 requested a full tablet report, sending...
I20260812 11:23:51.542295 16888 ts_manager.cc:194] Registered new tserver with Master: e1a4cc7cb3b44516b1a0a75ad05b628d (127.12.110.194:41207)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
I20260812 11:23:51.543535 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:40947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:23:51.576143 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:51.926549 17426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:51.926823 17426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:51.926998 17426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:51.934507 17426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:51.934650 17426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:51.934836 17426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:23:51.946218 17426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37193
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:51.947561 17426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:51.948918 17426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:51.962783 17433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:51.962873 17437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:51.962810 17439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:51.963495 17426 server_base.cc:1034] running on GCE node
I20260812 11:23:51.964056 17426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:51.964906 17426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:51.966171 17426 hybrid_clock.cc:648] HybridClock initialized: now 1786533831966130 us; error 46 us; skew 500 ppm
I20260812 11:23:51.966509 17426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:51 dist-test-slave-h6n0 krb5kdc[16812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:51.969738 17426 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:23:51.971244 17426 webserver.cc:460] Webserver started at http://127.12.110.195:35793/ using document root <none> and password file <none>
I20260812 11:23:51.971884 17426 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:51.971988 17426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:51.972242 17426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:51.975144 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "9834ab9ac5474849b465763b2cd0fbb3"
format_stamp: "Formatted at 2026-08-12 11:23:51 on dist-test-slave-h6n0"
server_key: "a1d03333624de61d72798512507ad9d4"
server_key_iv: "ef6c601b2092c8d846e4c44e2bc6ed5d"
server_key_version: "encryptionkey@0"
I20260812 11:23:51.975878 17426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "9834ab9ac5474849b465763b2cd0fbb3"
format_stamp: "Formatted at 2026-08-12 11:23:51 on dist-test-slave-h6n0"
server_key: "a1d03333624de61d72798512507ad9d4"
server_key_iv: "ef6c601b2092c8d846e4c44e2bc6ed5d"
server_key_version: "encryptionkey@0"
I20260812 11:23:51.980890 17426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:23:51.984313 17452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:51.985721 17426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:23:51.985898 17426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "9834ab9ac5474849b465763b2cd0fbb3"
format_stamp: "Formatted at 2026-08-12 11:23:51 on dist-test-slave-h6n0"
server_key: "a1d03333624de61d72798512507ad9d4"
server_key_iv: "ef6c601b2092c8d846e4c44e2bc6ed5d"
server_key_version: "encryptionkey@0"
I20260812 11:23:51.986104 17426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:51.995069 17170 heartbeater.cc:503] Master 127.12.110.254:37193 was elected leader, sending a full tablet report...
I20260812 11:23:52.006861 17426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:52.010941 17426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:52.011232 17426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:52.012470 17426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:52.014019 17426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:52.014138 17426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:52.014317 17426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:52.014401 17426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:52.050855 17426 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:32769
I20260812 11:23:52.050913 17642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:32769 every 8 connection(s)
I20260812 11:23:52.052320 17426 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:23:52.056668 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17426
I20260812 11:23:52.056905 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:23:52.057542 12731 external_mini_cluster.cc:1428] Setting key 8bfa19194867cc375853af387a50f3fe
Aug 12 11:23:52 dist-test-slave-h6n0 krb5kdc[16812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533831, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:52.076344 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:52.055186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:40893 (local address 127.12.110.254:37193)
0812 11:23:52.055719 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:52.055725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:52.056855 (+ 1130us) server_negotiation.cc:407] Connection header received
0812 11:23:52.059372 (+ 2517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:52.059377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:52.059484 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:52.059603 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:52.061740 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:52.062694 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:52.064701 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:52.065137 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:52.065285 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:52.069564 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:52.069570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:52.069573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:52.069606 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:52.073163 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:52.073959 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:52.073966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:52.073972 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:52.074105 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:52.074578 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:52.074583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:52.074586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:52.074868 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:52.075029 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:52.075723 (+ 694us) server_negotiation.cc:299] Negotiation successful
0812 11:23:52.075946 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":244,"threads_started":1}
I20260812 11:23:52.077505 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:52.055694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37193 (local address 127.12.110.195:40893)
0812 11:23:52.056655 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:52.056709 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:52.059098 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:52.059888 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:52.059900 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:52.060901 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:52.061551 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:52.061570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:52.062869 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:52.062874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:52.064157 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:52.064170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:52.065395 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:52.065405 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:52.065670 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:52.066412 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:52.066441 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:52.069311 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:52.073326 (+ 4015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:52.073338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:52.073357 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:52.073814 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:52.074222 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:52.074227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:52.074231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:52.074427 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:52.075001 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:52.075009 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:52.075364 (+ 355us) client_negotiation.cc:769] Sending connection context
0812 11:23:52.075650 (+ 286us) client_negotiation.cc:240] Negotiation successful
0812 11:23:52.076219 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":302,"threads_started":1}
I20260812 11:23:52.079322 17643 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37193
I20260812 11:23:52.079675 17643 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:52.080451 17643 heartbeater.cc:511] Master 127.12.110.254:37193 requested a full tablet report, sending...
I20260812 11:23:52.082223 16888 ts_manager.cc:194] Registered new tserver with Master: 9834ab9ac5474849b465763b2cd0fbb3 (127.12.110.195:32769)
I20260812 11:23:52.083168 16888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:40893
I20260812 11:23:52.086891 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:23:52.135619 17646 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:23:52.132687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52862 (local address 127.12.110.254:37193)
0812 11:23:52.132949 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:52.132959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:52.133360 (+ 401us) server_negotiation.cc:407] Connection header received
0812 11:23:52.133507 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:52.133511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:52.133591 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:52.133679 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:52.135335 (+ 1656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52862: BlockingRecv error: recv got EOF from 127.0.0.1:52862 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:23:52.140164 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16980
W20260812 11:23:52.143596 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16980: Not authorized: Client connection negotiation failed: client connection to 127.12.110.193:33303: server requires authentication, but client does not have Kerberos credentials available
I20260812 11:23:52.143800 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16980
I20260812 11:23:52.171316 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17180
W20260812 11:23:52.174208 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17180: Not authorized: Client connection negotiation failed: client connection to 127.12.110.194:41207: server requires authentication, but client does not have Kerberos credentials available
I20260812 11:23:52.174382 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17180
I20260812 11:23:52.198335 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17426
W20260812 11:23:52.201627 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17426: Not authorized: Client connection negotiation failed: client connection to 127.12.110.195:32769: server requires authentication, but client does not have Kerberos credentials available
I20260812 11:23:52.201784 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17426
I20260812 11:23:52.229506 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16839
I20260812 11:23:52.231410 16933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:52.345814 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16839
2026-08-12T11:23:52Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4202 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:52 dist-test-slave-h6n0 krb5kdc[17670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:52 dist-test-slave-h6n0 krb5kdc[17670](info): set up 2 sockets
Aug 12 11:23:52 dist-test-slave-h6n0 krb5kdc[17670](info): commencing operation
krb5kdc: starting...
W20260812 11:23:53.745025 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:23:53 dist-test-slave-h6n0 krb5kdc[17670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:23:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:23:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:23:53.999100 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:39539
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:39539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:54.348116 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:54.348474 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:54.348639 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:54.355677 17690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:23:54.355821 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:54.355934 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:54.356042 17690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:23:54.356137 17690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:23:54.365711 17690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:39539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:39539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:54.367074 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:54.368420 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:54.380764 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:54.380770 17701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:54.380769 17709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:54.383054 17690 server_base.cc:1034] running on GCE node
I20260812 11:23:54.384359 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:54.386451 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:54.387790 17690 hybrid_clock.cc:648] HybridClock initialized: now 1786533834387758 us; error 88 us; skew 500 ppm
I20260812 11:23:54.388670 17690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:54 dist-test-slave-h6n0 krb5kdc[17670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533834, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:54.395277 17690 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:23:54.396958 17690 webserver.cc:460] Webserver started at http://127.12.110.254:39559/ using document root <none> and password file <none>
I20260812 11:23:54.397703 17690 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:54.397843 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:54.398202 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:54.401353 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "b80ed697aa7640858ae0a2681acc2625"
format_stamp: "Formatted at 2026-08-12 11:23:54 on dist-test-slave-h6n0"
server_key: "3719681b9dc08968db61e56a62392c9c"
server_key_iv: "33076130bab956968e7e22e4b96e82dd"
server_key_version: "encryptionkey@0"
I20260812 11:23:54.402161 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "b80ed697aa7640858ae0a2681acc2625"
format_stamp: "Formatted at 2026-08-12 11:23:54 on dist-test-slave-h6n0"
server_key: "3719681b9dc08968db61e56a62392c9c"
server_key_iv: "33076130bab956968e7e22e4b96e82dd"
server_key_version: "encryptionkey@0"
I20260812 11:23:54.407480 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260812 11:23:54.410941 17719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:54.412240 17690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:23:54.412441 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "b80ed697aa7640858ae0a2681acc2625"
format_stamp: "Formatted at 2026-08-12 11:23:54 on dist-test-slave-h6n0"
server_key: "3719681b9dc08968db61e56a62392c9c"
server_key_iv: "33076130bab956968e7e22e4b96e82dd"
server_key_version: "encryptionkey@0"
I20260812 11:23:54.412650 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:54.436914 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:54.441780 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:54.442093 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:54.461887 17690 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:39539
I20260812 11:23:54.462000 17801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:39539 every 8 connection(s)
I20260812 11:23:54.463311 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:23:54.464174 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17690
I20260812 11:23:54.464462 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:23:54.465166 12731 external_mini_cluster.cc:1428] Setting key 1d334231b7eaa342f14bcf40481306b6
I20260812 11:23:54.467578 17802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 11:23:54 dist-test-slave-h6n0 krb5kdc[17670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:54.481447 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625: Bootstrap starting.
I20260812 11:23:54.485832 17802 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:54.487123 17802 log.cc:824] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:54.492055 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:54.467035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.12.110.254:39539)
0812 11:23:54.467930 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:54.467941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:54.467980 (+ 39us) server_negotiation.cc:407] Connection header received
0812 11:23:54.469657 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:54.469677 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:54.470043 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:54.470389 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:54.472375 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:54.473761 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:54.475620 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:54.476315 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:54.476452 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:54.481445 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:54.481456 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:54.481477 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:54.481532 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:54.485541 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:54.486353 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:54.486362 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:54.486370 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:54.486604 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:54.487004 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:54.487010 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:54.487013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:54.487963 (+ 950us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:54.488354 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:54.489549 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0812 11:23:54.490239 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":407,"threads_started":1}
I20260812 11:23:54.492350 17802 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625: No bootstrap required, opened a new log
I20260812 11:23:54.497152 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:23:54.503404 17802 raft_consensus.cc:348] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80ed697aa7640858ae0a2681acc2625" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39539 } }
I20260812 11:23:54.503957 17802 raft_consensus.cc:374] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:54.504081 17802 raft_consensus.cc:729] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80ed697aa7640858ae0a2681acc2625, State: Initialized, Role: FOLLOWER
I20260812 11:23:54.504809 17802 consensus_queue.cc:260] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80ed697aa7640858ae0a2681acc2625" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39539 } }
I20260812 11:23:54.505151 17802 raft_consensus.cc:388] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:23:54.505311 17802 raft_consensus.cc:482] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:23:54.505517 17802 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:54.509210 17802 raft_consensus.cc:504] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80ed697aa7640858ae0a2681acc2625" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39539 } }
I20260812 11:23:54.509895 17802 leader_election.cc:302] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b80ed697aa7640858ae0a2681acc2625; no voters:
I20260812 11:23:54.510704 17802 leader_election.cc:288] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:23:54.511121 17810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:54.512590 17810 raft_consensus.cc:686] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [term 1 LEADER]: Becoming Leader. State: Replica: b80ed697aa7640858ae0a2681acc2625, State: Running, Role: LEADER
I20260812 11:23:54.513420 17810 consensus_queue.cc:237] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80ed697aa7640858ae0a2681acc2625" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39539 } }
I20260812 11:23:54.513870 17802 sys_catalog.cc:564] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:23:54.522742 17811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b80ed697aa7640858ae0a2681acc2625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80ed697aa7640858ae0a2681acc2625" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39539 } } }
I20260812 11:23:54.523164 17811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:54.523818 17817 sys_catalog.cc:455] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b80ed697aa7640858ae0a2681acc2625. Latest consensus state: current_term: 1 leader_uuid: "b80ed697aa7640858ae0a2681acc2625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b80ed697aa7640858ae0a2681acc2625" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39539 } } }
I20260812 11:23:54.524166 17817 sys_catalog.cc:458] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625 [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:54.534029 17826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:23:54.543391 17826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:23:54.555171 17826 catalog_manager.cc:1347] Generated new cluster ID: 96eda4765f95464492fb027a61f70110
I20260812 11:23:54.555366 17826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:23:54.582345 17826 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:23:54.583087 17807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:54.572123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.12.110.254:39539)
0812 11:23:54.572312 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:54.572325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:54.573009 (+ 684us) server_negotiation.cc:407] Connection header received
0812 11:23:54.573218 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:54.573224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:54.573307 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:54.573406 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:54.574841 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:54.575988 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:54.577954 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:54.578694 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:54.578836 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:54.581014 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:54.581033 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:54.581037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:23:54.581869 (+ 832us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:54.582096 (+ 227us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:23:54.582139 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:54.582293 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:54.582479 (+ 186us) server_negotiation.cc:299] Negotiation successful
0812 11:23:54.582628 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":42}
I20260812 11:23:54.584471 17826 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:23:54.594231 17826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b80ed697aa7640858ae0a2681acc2625: Generated new TSK 0
I20260812 11:23:54.595139 17826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:23:54.610425 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17690
I20260812 11:23:54.612166 17791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:54.707288 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17690
2026-08-12T11:23:54Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2349 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:54 dist-test-slave-h6n0 krb5kdc[17859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:54 dist-test-slave-h6n0 krb5kdc[17859](info): set up 2 sockets
Aug 12 11:23:54 dist-test-slave-h6n0 krb5kdc[17859](info): commencing operation
krb5kdc: starting...
W20260812 11:23:56.125065 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.369s user 0.001s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:23:56 dist-test-slave-h6n0 krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:23:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:23:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:23:56.382627 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:32985
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:56.736459 17884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:56.736739 17884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:56.736836 17884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:56.744151 17884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:23:56.744267 17884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:56.744315 17884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:56.744374 17884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:23:56.744412 17884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:23:56.753881 17884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:32985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:56.755103 17884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:56.756345 17884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:56.768973 17890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:56.769012 17894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:56.768963 17891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:56.769862 17884 server_base.cc:1034] running on GCE node
I20260812 11:23:56.770774 17884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:56.772217 17884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:56.773461 17884 hybrid_clock.cc:648] HybridClock initialized: now 1786533836773413 us; error 52 us; skew 500 ppm
I20260812 11:23:56.773816 17884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:56 dist-test-slave-h6n0 krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533836, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:56.777194 17884 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:23:56.778775 17884 webserver.cc:460] Webserver started at http://127.12.110.254:33883/ using document root <none> and password file <none>
I20260812 11:23:56.779565 17884 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:56.779734 17884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:56.780624 17884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:56.784510 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "ef2bfcc1b7a04c48845b03ffb26073be"
format_stamp: "Formatted at 2026-08-12 11:23:56 on dist-test-slave-h6n0"
server_key: "8d3fd71141f278ff76256fa70670b183"
server_key_iv: "42b3609b0670250647d2f3f57addc64b"
server_key_version: "encryptionkey@0"
I20260812 11:23:56.785408 17884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "ef2bfcc1b7a04c48845b03ffb26073be"
format_stamp: "Formatted at 2026-08-12 11:23:56 on dist-test-slave-h6n0"
server_key: "8d3fd71141f278ff76256fa70670b183"
server_key_iv: "42b3609b0670250647d2f3f57addc64b"
server_key_version: "encryptionkey@0"
I20260812 11:23:56.791247 17884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 11:23:56.796010 17905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:56.797389 17884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:23:56.797593 17884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "ef2bfcc1b7a04c48845b03ffb26073be"
format_stamp: "Formatted at 2026-08-12 11:23:56 on dist-test-slave-h6n0"
server_key: "8d3fd71141f278ff76256fa70670b183"
server_key_iv: "42b3609b0670250647d2f3f57addc64b"
server_key_version: "encryptionkey@0"
I20260812 11:23:56.797798 17884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:56.823307 17884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:56.829254 17884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:56.829636 17884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:56.853492 17884 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:32985
I20260812 11:23:56.853538 17977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:32985 every 8 connection(s)
I20260812 11:23:56.855041 17884 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:23:56.858888 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17884
I20260812 11:23:56.859066 17978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:23:56.859128 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:23:56.859970 12731 external_mini_cluster.cc:1428] Setting key a715fd3b6bd852d55c0f458d2c5a9ba9
I20260812 11:23:56.873634 17978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be: Bootstrap starting.
Aug 12 11:23:56 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:56.877350 17978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be: Neither blocks nor log segments found. Creating new log.
I20260812 11:23:56.878664 17978 log.cc:824] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be: Log is configured to *not* fsync() on all Append() calls
I20260812 11:23:56.881690 17978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be: No bootstrap required, opened a new log
I20260812 11:23:56.884560 17981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:56.862191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.12.110.254:32985)
0812 11:23:56.863102 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:56.863115 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:56.863159 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:23:56.865572 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:56.865595 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:56.866013 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:56.866342 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:56.868145 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:56.869469 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:56.871324 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:56.871998 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:56.872133 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:56.877095 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:56.877109 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:56.877132 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:56.877189 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:56.880538 (+ 3349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:56.881249 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:56.881255 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:56.881261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:56.881425 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:56.881819 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:56.881828 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:56.881831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:56.882332 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:56.882720 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:56.883111 (+ 391us) server_negotiation.cc:299] Negotiation successful
0812 11:23:56.883564 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":455,"threads_started":1}
I20260812 11:23:56.890831 17978 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32985 } }
I20260812 11:23:56.891289 17978 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:23:56.891407 17978 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2bfcc1b7a04c48845b03ffb26073be, State: Initialized, Role: FOLLOWER
I20260812 11:23:56.892053 17978 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32985 } }
I20260812 11:23:56.892333 17978 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:23:56.892465 17978 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:23:56.892590 17978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:23:56.894855 17978 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32985 } }
I20260812 11:23:56.895383 17978 leader_election.cc:302] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef2bfcc1b7a04c48845b03ffb26073be; no voters:
I20260812 11:23:56.895954 17978 leader_election.cc:288] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:23:56.896101 17983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:23:56.896799 17983 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [term 1 LEADER]: Becoming Leader. State: Replica: ef2bfcc1b7a04c48845b03ffb26073be, State: Running, Role: LEADER
I20260812 11:23:56.897404 17983 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32985 } }
I20260812 11:23:56.898062 17978 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:23:56.900032 17984 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32985 } } }
I20260812 11:23:56.900403 17984 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:56.900524 17985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2bfcc1b7a04c48845b03ffb26073be. Latest consensus state: current_term: 1 leader_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2bfcc1b7a04c48845b03ffb26073be" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 32985 } } }
I20260812 11:23:56.900906 17985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be [sys.catalog]: This master's current role is: LEADER
I20260812 11:23:56.903532 17988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:23:56.909876 17988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:23:56.918023 17988 catalog_manager.cc:1347] Generated new cluster ID: ffe6a150891248b3a259d0b84f744aa5
I20260812 11:23:56.918148 17988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:23:56.949915 17988 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:23:56.951463 17988 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:23:56.961457 17988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef2bfcc1b7a04c48845b03ffb26073be: Generated new TSK 0
I20260812 11:23:56.963158 17988 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:23:57.035301 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32985
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:57.397742 18016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:57.398041 18016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:57.398178 18016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:57.405532 18016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:57.405640 18016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:57.405751 18016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:23:57.415446 18016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:57.416656 18016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:57.417934 18016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:57.432165 18023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:57.432096 18030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:57.432189 18026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:57.433046 18016 server_base.cc:1034] running on GCE node
I20260812 11:23:57.433665 18016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:57.434644 18016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:57.435864 18016 hybrid_clock.cc:648] HybridClock initialized: now 1786533837435833 us; error 42 us; skew 500 ppm
I20260812 11:23:57.436180 18016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:57 dist-test-slave-h6n0 krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:57.439715 18016 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:23:57.441190 18016 webserver.cc:460] Webserver started at http://127.12.110.193:42043/ using document root <none> and password file <none>
I20260812 11:23:57.441855 18016 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:57.442001 18016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:57.442305 18016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:57.445215 18016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "ade30a87c9f5444ba756c37a438010a8"
format_stamp: "Formatted at 2026-08-12 11:23:57 on dist-test-slave-h6n0"
server_key: "eebb3fdab5a12f83d8c70b603cc10d3c"
server_key_iv: "2b6a4bb06a8acef27d6e98c93286257b"
server_key_version: "encryptionkey@0"
I20260812 11:23:57.445991 18016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "ade30a87c9f5444ba756c37a438010a8"
format_stamp: "Formatted at 2026-08-12 11:23:57 on dist-test-slave-h6n0"
server_key: "eebb3fdab5a12f83d8c70b603cc10d3c"
server_key_iv: "2b6a4bb06a8acef27d6e98c93286257b"
server_key_version: "encryptionkey@0"
I20260812 11:23:57.451073 18016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:23:57.454805 18037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:57.456194 18016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:23:57.456380 18016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "ade30a87c9f5444ba756c37a438010a8"
format_stamp: "Formatted at 2026-08-12 11:23:57 on dist-test-slave-h6n0"
server_key: "eebb3fdab5a12f83d8c70b603cc10d3c"
server_key_iv: "2b6a4bb06a8acef27d6e98c93286257b"
server_key_version: "encryptionkey@0"
I20260812 11:23:57.456593 18016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:57.471789 18016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:57.475975 18016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:57.476269 18016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:57.477247 18016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:57.478854 18016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:57.479071 18016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:57.479379 18016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:57.479468 18016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:57.519671 18016 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:46309
I20260812 11:23:57.519731 18219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:46309 every 8 connection(s)
I20260812 11:23:57.520992 18016 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:23:57.522323 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18016
I20260812 11:23:57.522578 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:23:57.523456 12731 external_mini_cluster.cc:1428] Setting key c49115f09f8b05a9f2ed214a16eb2716
Aug 12 11:23:57 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:57.542842 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:57.523954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:59459 (local address 127.12.110.254:32985)
0812 11:23:57.524478 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:57.524485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:57.525868 (+ 1383us) server_negotiation.cc:407] Connection header received
0812 11:23:57.528092 (+ 2224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:57.528100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:57.528190 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:57.528304 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:57.529723 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:57.530909 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:57.532479 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:57.533210 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:57.533364 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:57.537018 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:57.537023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:57.537026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:57.537056 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:57.539780 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:57.540499 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:57.540506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:57.540510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:57.540632 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:57.541061 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:57.541066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:57.541069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:57.541345 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:57.541502 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:57.542208 (+ 706us) server_negotiation.cc:299] Negotiation successful
0812 11:23:57.542452 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":273,"threads_started":1}
I20260812 11:23:57.543805 18224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:57.524780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32985 (local address 127.12.110.193:59459)
0812 11:23:57.525681 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:57.525740 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:57.527795 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:57.528516 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:57.528525 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:57.529018 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:57.529560 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:57.529577 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:57.531074 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:57.531079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:57.532315 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:57.532329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:57.533451 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:57.533456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:57.533655 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:57.534347 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:57.534369 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:57.536805 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:57.539925 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:57.539935 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:57.539950 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:57.540371 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:57.540744 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:57.540749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:57.540752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:57.540949 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:57.541498 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:57.541507 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:57.541842 (+ 335us) client_negotiation.cc:769] Sending connection context
0812 11:23:57.542104 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 11:23:57.542665 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":391,"threads_started":1}
I20260812 11:23:57.546198 18221 heartbeater.cc:347] Connected to a master server at 127.12.110.254:32985
I20260812 11:23:57.546605 18221 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:57.547643 18221 heartbeater.cc:511] Master 127.12.110.254:32985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
I20260812 11:23:57.550649 17931 ts_manager.cc:194] Registered new tserver with Master: ade30a87c9f5444ba756c37a438010a8 (127.12.110.193:46309)
I20260812 11:23:57.552354 17931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:59459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:23:57.587931 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32985
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:57.932873 18230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:57.933220 18230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:57.933409 18230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:57.940805 18230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:57.940966 18230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:57.941154 18230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:23:57.950999 18230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:57.952339 18230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:57.953672 18230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:57.969517 18238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:57.969533 18240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:57.969564 18243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:57.970348 18230 server_base.cc:1034] running on GCE node
I20260812 11:23:57.971032 18230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:57.971997 18230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:57.973223 18230 hybrid_clock.cc:648] HybridClock initialized: now 1786533837973185 us; error 59 us; skew 500 ppm
I20260812 11:23:57.973555 18230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:57 dist-test-slave-h6n0 krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:57.976806 18230 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:23:57.978283 18230 webserver.cc:460] Webserver started at http://127.12.110.194:34069/ using document root <none> and password file <none>
I20260812 11:23:57.979017 18230 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:57.979164 18230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:57.979480 18230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:57.982695 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "0fe9b9e5b6e14149a01311410cca11c5"
format_stamp: "Formatted at 2026-08-12 11:23:57 on dist-test-slave-h6n0"
server_key: "ea9fe31f712970730035f4efe8e75279"
server_key_iv: "a21a4ed541cb5f3faa79b42c47c802b4"
server_key_version: "encryptionkey@0"
I20260812 11:23:57.983554 18230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "0fe9b9e5b6e14149a01311410cca11c5"
format_stamp: "Formatted at 2026-08-12 11:23:57 on dist-test-slave-h6n0"
server_key: "ea9fe31f712970730035f4efe8e75279"
server_key_iv: "a21a4ed541cb5f3faa79b42c47c802b4"
server_key_version: "encryptionkey@0"
I20260812 11:23:57.988900 18230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:23:57.992362 18258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:57.993655 18230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:23:57.993845 18230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "0fe9b9e5b6e14149a01311410cca11c5"
format_stamp: "Formatted at 2026-08-12 11:23:57 on dist-test-slave-h6n0"
server_key: "ea9fe31f712970730035f4efe8e75279"
server_key_iv: "a21a4ed541cb5f3faa79b42c47c802b4"
server_key_version: "encryptionkey@0"
I20260812 11:23:57.994056 18230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:58.007880 18230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:58.012735 18230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:58.013108 18230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:58.014277 18230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:58.015861 18230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:58.015975 18230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:58.016136 18230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:58.016218 18230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:58.053256 18230 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:37765
I20260812 11:23:58.053279 18442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:37765 every 8 connection(s)
I20260812 11:23:58.054677 18230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:23:58.055131 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18230
I20260812 11:23:58.055486 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:23:58.056154 12731 external_mini_cluster.cc:1428] Setting key c0b5c9355b035a592a1fdec5c2cd7853
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533837, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:58.080231 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.057911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:54337 (local address 127.12.110.254:32985)
0812 11:23:58.058401 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:58.058408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:58.059686 (+ 1278us) server_negotiation.cc:407] Connection header received
0812 11:23:58.061892 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:58.061900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:58.061995 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:58.062131 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:58.063716 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.064867 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.067085 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.067815 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.067966 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.072473 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:58.072484 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:58.072493 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:58.072547 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:58.076435 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.077253 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.077261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.077267 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.077440 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.077895 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.077902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.077908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.078334 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:58.078574 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:58.079453 (+ 879us) server_negotiation.cc:299] Negotiation successful
0812 11:23:58.079741 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260812 11:23:58.081161 18446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.058608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32985 (local address 127.12.110.194:54337)
0812 11:23:58.059507 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:58.059578 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:58.061630 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:58.062365 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:58.062373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:58.062990 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:58.063545 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:58.063563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.065039 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.065044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:58.066851 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:58.066881 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.068099 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.068106 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:58.068392 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.069423 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:58.069460 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:58.072196 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:58.076635 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:58.076643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:58.076656 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:58.077106 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:58.077560 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:58.077564 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:58.077566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:58.077757 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:58.078684 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:58.078692 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:58.079043 (+ 351us) client_negotiation.cc:769] Sending connection context
0812 11:23:58.079286 (+ 243us) client_negotiation.cc:240] Negotiation successful
0812 11:23:58.079753 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":404,"threads_started":1}
I20260812 11:23:58.082911 18443 heartbeater.cc:347] Connected to a master server at 127.12.110.254:32985
I20260812 11:23:58.083271 18443 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:58.084252 18443 heartbeater.cc:511] Master 127.12.110.254:32985 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
I20260812 11:23:58.086181 17931 ts_manager.cc:194] Registered new tserver with Master: 0fe9b9e5b6e14149a01311410cca11c5 (127.12.110.194:37765)
I20260812 11:23:58.087172 17931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:54337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:23:58.120426 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32985
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:23:58.493415 18454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:23:58.493734 18454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:23:58.493834 18454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:23:58.503019 18454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:23:58.503233 18454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:23:58.503379 18454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:23:58.515139 18454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:32985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:23:58.516486 18454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:23:58.517755 18454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:23:58.533340 18461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:58.533423 18463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:23:58.533370 18465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:23:58.534211 18454 server_base.cc:1034] running on GCE node
I20260812 11:23:58.534951 18454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:23:58.535971 18454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:23:58.537207 18454 hybrid_clock.cc:648] HybridClock initialized: now 1786533838537167 us; error 61 us; skew 500 ppm
I20260812 11:23:58.537531 18454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533838, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:23:58.540732 18454 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:23:58.542518 18454 webserver.cc:460] Webserver started at http://127.12.110.195:39521/ using document root <none> and password file <none>
I20260812 11:23:58.543241 18454 fs_manager.cc:359] Metadata directory not provided
I20260812 11:23:58.543349 18454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:23:58.543591 18454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:23:58.546422 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "32c58c6bd37742f1a197cf7520a27c1b"
format_stamp: "Formatted at 2026-08-12 11:23:58 on dist-test-slave-h6n0"
server_key: "76536dfad94f923faf731d14fa68080c"
server_key_iv: "993efe48f7f38401cbfcbd416a5a8428"
server_key_version: "encryptionkey@0"
I20260812 11:23:58.547222 18454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "32c58c6bd37742f1a197cf7520a27c1b"
format_stamp: "Formatted at 2026-08-12 11:23:58 on dist-test-slave-h6n0"
server_key: "76536dfad94f923faf731d14fa68080c"
server_key_iv: "993efe48f7f38401cbfcbd416a5a8428"
server_key_version: "encryptionkey@0"
I20260812 11:23:58.553241 18454 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 11:23:58.556777 18221 heartbeater.cc:503] Master 127.12.110.254:32985 was elected leader, sending a full tablet report...
I20260812 11:23:58.556835 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:58.558612 18454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:23:58.558799 18454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "32c58c6bd37742f1a197cf7520a27c1b"
format_stamp: "Formatted at 2026-08-12 11:23:58 on dist-test-slave-h6n0"
server_key: "76536dfad94f923faf731d14fa68080c"
server_key_iv: "993efe48f7f38401cbfcbd416a5a8428"
server_key_version: "encryptionkey@0"
I20260812 11:23:58.559049 18454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:23:58.585616 18454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:23:58.591655 18454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:23:58.591951 18454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:23:58.592948 18454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:23:58.594328 18454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:23:58.594410 18454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:23:58.594496 18454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:23:58.594616 18454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 11:23:58.634123 18454 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:40085
I20260812 11:23:58.634162 18662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:40085 every 8 connection(s)
I20260812 11:23:58.635497 18454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:23:58.640877 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18454
I20260812 11:23:58.641074 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:23:58.641657 12731 external_mini_cluster.cc:1428] Setting key 5c7947d0f365b8158559373ed0422226
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533838, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:58.655577 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.638295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:50419 (local address 127.12.110.254:32985)
0812 11:23:58.638837 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:58.638842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:58.639641 (+ 799us) server_negotiation.cc:407] Connection header received
0812 11:23:58.641495 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:58.641503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:58.641587 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:58.641720 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:58.643317 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.644285 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.645909 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.646466 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.646616 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.649788 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:58.649803 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:58.649806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:58.649838 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:58.652470 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.653136 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.653139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.653142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.653268 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.653643 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.653646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.653648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.654014 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:58.654148 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:58.654955 (+ 807us) server_negotiation.cc:299] Negotiation successful
0812 11:23:58.655181 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":284,"threads_started":1}
I20260812 11:23:58.656808 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.638812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:32985 (local address 127.12.110.195:50419)
0812 11:23:58.639477 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0812 11:23:58.639513 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 11:23:58.641231 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:23:58.641947 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:23:58.641955 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:23:58.642432 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:23:58.643154 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:58.643172 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.644494 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.644500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:58.645755 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:23:58.645767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.646692 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.646698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:23:58.646882 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.647487 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:23:58.647507 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:23:58.649589 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:23:58.652599 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:58.652605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:58.652615 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:58.653025 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:58.653370 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:23:58.653373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:23:58.653375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:23:58.653555 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:23:58.654155 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:23:58.654161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:23:58.654432 (+ 271us) client_negotiation.cc:769] Sending connection context
0812 11:23:58.654779 (+ 347us) client_negotiation.cc:240] Negotiation successful
0812 11:23:58.655368 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":258,"threads_started":1}
I20260812 11:23:58.658272 18663 heartbeater.cc:347] Connected to a master server at 127.12.110.254:32985
I20260812 11:23:58.658624 18663 heartbeater.cc:464] Registering TS with master...
I20260812 11:23:58.659332 18663 heartbeater.cc:511] Master 127.12.110.254:32985 requested a full tablet report, sending...
I20260812 11:23:58.660750 17931 ts_manager.cc:194] Registered new tserver with Master: 32c58c6bd37742f1a197cf7520a27c1b (127.12.110.195:40085)
I20260812 11:23:58.661736 17931 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:50419
I20260812 11:23:58.669497 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533838, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533838, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:23:58.736891 18670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.719025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.12.110.254:32985)
0812 11:23:58.719235 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:58.719240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:58.719916 (+ 676us) server_negotiation.cc:407] Connection header received
0812 11:23:58.720080 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:58.720085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:58.720359 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:58.720478 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:58.722451 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.723276 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.724895 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.725472 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.725624 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.730894 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:58.730901 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:58.730905 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:58.730963 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:58.734304 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.735058 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.735062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.735065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.735212 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.735618 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.735621 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.735623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.735993 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:58.736186 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:58.736486 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 11:23:58.736669 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260812 11:23:58.737488 17931 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47980
I20260812 11:23:58.739392 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18016
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533838, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:23:58.755904 18685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.740577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52372 (local address 127.12.110.193:46309)
0812 11:23:58.741187 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:58.741194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:58.741218 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:23:58.741273 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:58.741279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:58.741478 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:58.741604 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:58.742928 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.743998 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.745158 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.745719 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.745820 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.749827 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:58.749840 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:58.749854 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:58.749888 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:58.752831 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.753529 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.753536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.753539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.753696 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.754148 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.754152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.754154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.754606 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:58.754954 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:58.755254 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 11:23:58.755544 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":328,"threads_started":1}
W20260812 11:23:58.757436 18210 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52372
W20260812 11:23:58.758610 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18016: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:58.758846 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18016
I20260812 11:23:58.787952 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18230
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533838, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:23:58.804602 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.788968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60352 (local address 127.12.110.194:37765)
0812 11:23:58.789528 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:58.789535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:58.789562 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:23:58.789625 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:58.789631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:58.789851 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:58.789990 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:58.791381 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.792418 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.793654 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.794186 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.794283 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.798666 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:58.798681 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:58.798697 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:58.798737 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:58.801733 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.802401 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.802409 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.802427 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.802617 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.802991 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.802996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.803000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.803341 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:58.803676 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:58.803976 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 11:23:58.804245 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":305,"threads_started":1}
W20260812 11:23:58.805792 18433 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60352
W20260812 11:23:58.806991 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18230: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:58.807125 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18230
I20260812 11:23:58.833335 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18454
Aug 12 11:23:58 dist-test-slave-h6n0 krb5kdc[17859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533838, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:23:58.853300 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:23:58.834383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.12.110.195:40085)
0812 11:23:58.835156 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0812 11:23:58.835164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:23:58.835198 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:23:58.835265 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:23:58.835272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:23:58.835506 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:23:58.835714 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:23:58.837407 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.838664 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.840024 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:23:58.840644 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:23:58.840807 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:23:58.845450 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:23:58.845463 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:23:58.845478 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:23:58.845516 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:23:58.848518 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.849757 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.849768 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.849770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.849977 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:23:58.851008 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:23:58.851013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:23:58.851016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:23:58.851424 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:23:58.851670 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0812 11:23:58.852167 (+ 497us) server_negotiation.cc:299] Negotiation successful
0812 11:23:58.852676 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":414,"threads_started":1}
W20260812 11:23:58.855140 18656 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45114
W20260812 11:23:58.856665 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18454: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:23:58.856945 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18454
I20260812 11:23:58.881240 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17884
I20260812 11:23:58.882977 17972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:23:58.996097 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17884
2026-08-12T11:23:59Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4295 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:23:59 dist-test-slave-h6n0 krb5kdc[18694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:23:59 dist-test-slave-h6n0 krb5kdc[18694](info): set up 2 sockets
Aug 12 11:23:59 dist-test-slave-h6n0 krb5kdc[18694](info): commencing operation
krb5kdc: starting...
W20260812 11:24:00.387342 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.336s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:24:00 dist-test-slave-h6n0 krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:24:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:24:00.670951 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41995
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:41995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:01.028286 18714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:01.028589 18714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:01.028739 18714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:01.036089 18714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:01.036211 18714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:01.036330 18714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:01.036433 18714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:01.036525 18714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:01.046429 18714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:41995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:01.047781 18714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:01.049185 18714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:01.061102 18729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:01.061354 18725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:01.061326 18723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:01.062356 18714 server_base.cc:1034] running on GCE node
I20260812 11:24:01.063618 18714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:01.065131 18714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:01.066381 18714 hybrid_clock.cc:648] HybridClock initialized: now 1786533841066328 us; error 52 us; skew 500 ppm
I20260812 11:24:01.066813 18714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:01 dist-test-slave-h6n0 krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533841, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:01.074216 18714 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:24:01.076363 18714 webserver.cc:460] Webserver started at http://127.12.110.254:37245/ using document root <none> and password file <none>
I20260812 11:24:01.077136 18714 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:01.077298 18714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:01.077629 18714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:01.081094 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "ed00c072ab5949df89e61f8a3880eba3"
format_stamp: "Formatted at 2026-08-12 11:24:01 on dist-test-slave-h6n0"
server_key: "71496d32fc50b16fd4c88ac14a3ed2f0"
server_key_iv: "4ea66ac7bb3a3d6b91862745e8b370d5"
server_key_version: "encryptionkey@0"
I20260812 11:24:01.082001 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "ed00c072ab5949df89e61f8a3880eba3"
format_stamp: "Formatted at 2026-08-12 11:24:01 on dist-test-slave-h6n0"
server_key: "71496d32fc50b16fd4c88ac14a3ed2f0"
server_key_iv: "4ea66ac7bb3a3d6b91862745e8b370d5"
server_key_version: "encryptionkey@0"
I20260812 11:24:01.087885 18714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260812 11:24:01.092350 18745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:01.093868 18714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:24:01.094105 18714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "ed00c072ab5949df89e61f8a3880eba3"
format_stamp: "Formatted at 2026-08-12 11:24:01 on dist-test-slave-h6n0"
server_key: "71496d32fc50b16fd4c88ac14a3ed2f0"
server_key_iv: "4ea66ac7bb3a3d6b91862745e8b370d5"
server_key_version: "encryptionkey@0"
I20260812 11:24:01.094340 18714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:01.112262 18714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:01.117367 18714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:01.117717 18714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:01.141321 18714 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:41995
I20260812 11:24:01.141338 18832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:41995 every 8 connection(s)
I20260812 11:24:01.142940 18714 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:01.146714 18833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:01.149214 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18714
I20260812 11:24:01.149456 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:01.150310 12731 external_mini_cluster.cc:1428] Setting key 5b634718d67a9b45fee2a0eb6014f8da
I20260812 11:24:01.159512 18833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3: Bootstrap starting.
I20260812 11:24:01.163447 18833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:01.164808 18833 log.cc:824] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:24:01 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:01.167989 18833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3: No bootstrap required, opened a new log
I20260812 11:24:01.173836 18838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:01.152428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.12.110.254:41995)
0812 11:24:01.153233 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:01.153245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:01.153292 (+ 47us) server_negotiation.cc:407] Connection header received
0812 11:24:01.155056 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:01.155077 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:01.155411 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:01.155728 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:01.157528 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:01.158923 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:01.160634 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:01.161581 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:01.161749 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:01.166744 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:01.166751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:01.166764 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:01.166798 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:01.169943 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:01.170758 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:01.170764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:01.170769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:01.170949 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:01.171342 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:01.171345 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:01.171347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:01.171833 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:01.172070 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:01.172482 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 11:24:01.172833 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":371,"threads_started":1}
I20260812 11:24:01.177254 18833 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed00c072ab5949df89e61f8a3880eba3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41995 } }
I20260812 11:24:01.177601 18833 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:01.177706 18833 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed00c072ab5949df89e61f8a3880eba3, State: Initialized, Role: FOLLOWER
I20260812 11:24:01.178241 18833 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed00c072ab5949df89e61f8a3880eba3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41995 } }
I20260812 11:24:01.178483 18833 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:01.178639 18833 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:01.178761 18833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:01.181047 18833 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed00c072ab5949df89e61f8a3880eba3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41995 } }
I20260812 11:24:01.181500 18833 leader_election.cc:302] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed00c072ab5949df89e61f8a3880eba3; no voters:
I20260812 11:24:01.182019 18833 leader_election.cc:288] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:01.182156 18842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:01.182883 18842 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [term 1 LEADER]: Becoming Leader. State: Replica: ed00c072ab5949df89e61f8a3880eba3, State: Running, Role: LEADER
I20260812 11:24:01.183889 18842 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed00c072ab5949df89e61f8a3880eba3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41995 } }
I20260812 11:24:01.184544 18833 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:01.187017 18843 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed00c072ab5949df89e61f8a3880eba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed00c072ab5949df89e61f8a3880eba3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41995 } } }
I20260812 11:24:01.187485 18843 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:01.187991 18844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed00c072ab5949df89e61f8a3880eba3. Latest consensus state: current_term: 1 leader_uuid: "ed00c072ab5949df89e61f8a3880eba3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed00c072ab5949df89e61f8a3880eba3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41995 } } }
I20260812 11:24:01.188357 18844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:01.191349 18853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:01.197503 18853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:01.207020 18853 catalog_manager.cc:1347] Generated new cluster ID: 7ddfb6e9acb747c0989b7af64acb01c1
I20260812 11:24:01.207142 18853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:01.226893 18853 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:01.229053 18853 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:01.248258 18853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed00c072ab5949df89e61f8a3880eba3: Generated new TSK 0
I20260812 11:24:01.249429 18853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:24:01.315266 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41995
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:01.664028 18873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:01.664379 18873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:01.664557 18873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:01.672742 18873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:01.672905 18873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:01.673080 18873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:01.683449 18873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:01.685074 18873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:01.686661 18873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:01.701220 18879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:01.701301 18880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:01.701273 18883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:01.702495 18873 server_base.cc:1034] running on GCE node
I20260812 11:24:01.703184 18873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:01.704424 18873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:01.705683 18873 hybrid_clock.cc:648] HybridClock initialized: now 1786533841705610 us; error 69 us; skew 500 ppm
I20260812 11:24:01.706073 18873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:01 dist-test-slave-h6n0 krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533841, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:01.710711 18873 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:24:01.712514 18873 webserver.cc:460] Webserver started at http://127.12.110.193:34043/ using document root <none> and password file <none>
I20260812 11:24:01.713287 18873 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:01.713418 18873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:01.713769 18873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:01.716925 18873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "434ab20d547c4bc0aea4205987ae2e36"
format_stamp: "Formatted at 2026-08-12 11:24:01 on dist-test-slave-h6n0"
server_key: "2231d09d48b2ea017407c3fcce79499a"
server_key_iv: "9f218f2d37de17c3814a9d6e3be18d43"
server_key_version: "encryptionkey@0"
I20260812 11:24:01.717746 18873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "434ab20d547c4bc0aea4205987ae2e36"
format_stamp: "Formatted at 2026-08-12 11:24:01 on dist-test-slave-h6n0"
server_key: "2231d09d48b2ea017407c3fcce79499a"
server_key_iv: "9f218f2d37de17c3814a9d6e3be18d43"
server_key_version: "encryptionkey@0"
I20260812 11:24:01.723658 18873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:24:01.727772 18897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:01.729818 18873 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:24:01.730019 18873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "434ab20d547c4bc0aea4205987ae2e36"
format_stamp: "Formatted at 2026-08-12 11:24:01 on dist-test-slave-h6n0"
server_key: "2231d09d48b2ea017407c3fcce79499a"
server_key_iv: "9f218f2d37de17c3814a9d6e3be18d43"
server_key_version: "encryptionkey@0"
I20260812 11:24:01.730235 18873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:01.743620 18873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:01.747624 18873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:01.747915 18873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:01.748891 18873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:01.750200 18873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:01.750304 18873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:01.750450 18873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:01.750558 18873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:01.785661 18873 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:35345
I20260812 11:24:01.785866 19066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:35345 every 8 connection(s)
I20260812 11:24:01.787071 18873 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:01.791905 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18873
I20260812 11:24:01.792145 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:24:01.792788 12731 external_mini_cluster.cc:1428] Setting key 081bfab76298c02b5e2de9d6e45363b0
Aug 12 11:24:01 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533841, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:01.810070 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:01.789911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:53245 (local address 127.12.110.254:41995)
0812 11:24:01.790387 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:01.790394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:01.791419 (+ 1025us) server_negotiation.cc:407] Connection header received
0812 11:24:01.793384 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:01.793390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:01.793479 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:01.793625 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:01.795599 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:01.796293 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:01.798956 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:01.799608 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:01.799842 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:01.804253 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:01.804259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:01.804262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:01.804304 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:01.806971 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:01.807726 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:01.807732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:01.807734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:01.807859 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:01.808304 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:01.808309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:01.808317 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:01.808579 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:01.808734 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:01.809468 (+ 734us) server_negotiation.cc:299] Negotiation successful
0812 11:24:01.809701 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":217,"threads_started":1}
I20260812 11:24:01.811120 19074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:01.790473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41995 (local address 127.12.110.193:53245)
0812 11:24:01.791259 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:01.791301 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:01.793080 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:01.793899 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:01.793907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:01.794610 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:01.795426 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:01.795452 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:01.796479 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:01.796488 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:01.798706 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:01.798725 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:01.800031 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:01.800042 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:01.800364 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:01.801253 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:01.801283 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:01.803977 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:01.807123 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:01.807132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:01.807149 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:01.807593 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:01.807970 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:01.807975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:01.807978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:01.808184 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:01.808711 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:01.808719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:01.809060 (+ 341us) client_negotiation.cc:769] Sending connection context
0812 11:24:01.809331 (+ 271us) client_negotiation.cc:240] Negotiation successful
0812 11:24:01.809875 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":286,"threads_started":1}
I20260812 11:24:01.812829 19068 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41995
I20260812 11:24:01.813376 19068 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:01.814127 19068 heartbeater.cc:511] Master 127.12.110.254:41995 requested a full tablet report, sending...
I20260812 11:24:01.816321 18770 ts_manager.cc:194] Registered new tserver with Master: 434ab20d547c4bc0aea4205987ae2e36 (127.12.110.193:35345)
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
I20260812 11:24:01.817790 18770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:53245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:24:01.851797 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41995
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:02.202049 19077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:02.202358 19077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:02.202569 19077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:02.210973 19077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:02.211093 19077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:02.211269 19077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:02.222338 19077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:02.223857 19077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:02.225240 19077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:02.239140 19083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:02.239120 19084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:02.239295 19088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:02.239957 19077 server_base.cc:1034] running on GCE node
I20260812 11:24:02.240609 19077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:02.241581 19077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:02.242810 19077 hybrid_clock.cc:648] HybridClock initialized: now 1786533842242749 us; error 45 us; skew 500 ppm
I20260812 11:24:02.243155 19077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:02 dist-test-slave-h6n0 krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:02.246778 19077 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:24:02.248421 19077 webserver.cc:460] Webserver started at http://127.12.110.194:43643/ using document root <none> and password file <none>
I20260812 11:24:02.249182 19077 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:02.249289 19077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:02.249583 19077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:02.252847 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "e221e54fa317429d9802e24974589b9c"
format_stamp: "Formatted at 2026-08-12 11:24:02 on dist-test-slave-h6n0"
server_key: "3bb652bf92125d2c5f5fcc80f852108e"
server_key_iv: "64d182a87dfa090ee8a43090040002e9"
server_key_version: "encryptionkey@0"
I20260812 11:24:02.253598 19077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "e221e54fa317429d9802e24974589b9c"
format_stamp: "Formatted at 2026-08-12 11:24:02 on dist-test-slave-h6n0"
server_key: "3bb652bf92125d2c5f5fcc80f852108e"
server_key_iv: "64d182a87dfa090ee8a43090040002e9"
server_key_version: "encryptionkey@0"
I20260812 11:24:02.258816 19077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 11:24:02.262169 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:02.263571 19077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:24:02.263751 19077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "e221e54fa317429d9802e24974589b9c"
format_stamp: "Formatted at 2026-08-12 11:24:02 on dist-test-slave-h6n0"
server_key: "3bb652bf92125d2c5f5fcc80f852108e"
server_key_iv: "64d182a87dfa090ee8a43090040002e9"
server_key_version: "encryptionkey@0"
I20260812 11:24:02.263954 19077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:02.282068 19077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:02.286216 19077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:02.286494 19077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:02.287468 19077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:02.289013 19077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:02.289113 19077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:02.289263 19077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:02.289336 19077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:02.325471 19077 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:45879
I20260812 11:24:02.325498 19268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:45879 every 8 connection(s)
I20260812 11:24:02.326997 19077 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:02.329185 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19077
I20260812 11:24:02.329421 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:24:02.329903 12731 external_mini_cluster.cc:1428] Setting key 119c7895b83877067575e6aad2783aa4
Aug 12 11:24:02 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:02.355957 19271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:02.330469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:50405 (local address 127.12.110.254:41995)
0812 11:24:02.331050 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:02.331056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:02.332337 (+ 1281us) server_negotiation.cc:407] Connection header received
0812 11:24:02.336121 (+ 3784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:02.336127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:02.336228 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:02.336378 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:02.338760 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.339574 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.342113 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.342913 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.343039 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:02.347420 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:02.347431 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:02.347439 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:02.347494 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:02.352274 (+ 4780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:02.353085 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:02.353093 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:02.353100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:02.353289 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:02.353759 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:02.353767 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:02.353773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:02.354203 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:02.354360 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:02.355137 (+ 777us) server_negotiation.cc:299] Negotiation successful
0812 11:24:02.355448 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
I20260812 11:24:02.356594 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:02.331248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41995 (local address 127.12.110.194:50405)
0812 11:24:02.332129 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:02.332178 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:02.335676 (+ 3498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:02.336714 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:02.336729 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:02.337604 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:02.338501 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:02.338566 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.339745 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.339753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:02.341927 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:02.341946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.343169 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.343179 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:02.343522 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:02.344630 (+ 1108us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:02.344654 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:02.347020 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:02.352425 (+ 5405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:02.352433 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:02.352447 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:02.352935 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:02.353414 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:02.353418 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:02.353421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:02.353620 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:02.354352 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:02.354358 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:02.354756 (+ 398us) client_negotiation.cc:769] Sending connection context
0812 11:24:02.354995 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 11:24:02.355403 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":385,"threads_started":1}
I20260812 11:24:02.358201 19269 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41995
I20260812 11:24:02.358516 19269 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:02.359337 19269 heartbeater.cc:511] Master 127.12.110.254:41995 requested a full tablet report, sending...
I20260812 11:24:02.360889 18770 ts_manager.cc:194] Registered new tserver with Master: e221e54fa317429d9802e24974589b9c (127.12.110.194:45879)
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
I20260812 11:24:02.362094 18770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:50405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:24:02.394567 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41995
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:02.765306 19275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:02.765647 19275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:02.765817 19275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:02.773391 19275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:02.773638 19275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:02.773993 19275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:02.784291 19275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:02.785916 19275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:02.787321 19275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:02.801148 19283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:02.801201 19284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:02.801221 19289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:02.802031 19275 server_base.cc:1034] running on GCE node
I20260812 11:24:02.802726 19275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:02.803666 19275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:02.804913 19275 hybrid_clock.cc:648] HybridClock initialized: now 1786533842804864 us; error 41 us; skew 500 ppm
I20260812 11:24:02.805276 19275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:02 dist-test-slave-h6n0 krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:02.808727 19275 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:24:02.810312 19275 webserver.cc:460] Webserver started at http://127.12.110.195:39941/ using document root <none> and password file <none>
I20260812 11:24:02.811059 19275 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:02.811203 19275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:02.811515 19275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:02.814427 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "c70eb5c6a0134d0cad8ef4a31a561faa"
format_stamp: "Formatted at 2026-08-12 11:24:02 on dist-test-slave-h6n0"
server_key: "f73dc584870348f345ec0adb6e18b899"
server_key_iv: "9efe356496045534b4588edceda3373b"
server_key_version: "encryptionkey@0"
I20260812 11:24:02.815316 19275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "c70eb5c6a0134d0cad8ef4a31a561faa"
format_stamp: "Formatted at 2026-08-12 11:24:02 on dist-test-slave-h6n0"
server_key: "f73dc584870348f345ec0adb6e18b899"
server_key_iv: "9efe356496045534b4588edceda3373b"
server_key_version: "encryptionkey@0"
I20260812 11:24:02.820515 19275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:24:02.820973 19068 heartbeater.cc:503] Master 127.12.110.254:41995 was elected leader, sending a full tablet report...
I20260812 11:24:02.824172 19298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:02.825876 19275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:24:02.826076 19275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "c70eb5c6a0134d0cad8ef4a31a561faa"
format_stamp: "Formatted at 2026-08-12 11:24:02 on dist-test-slave-h6n0"
server_key: "f73dc584870348f345ec0adb6e18b899"
server_key_iv: "9efe356496045534b4588edceda3373b"
server_key_version: "encryptionkey@0"
I20260812 11:24:02.826297 19275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:02.860217 19275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:02.864414 19275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:02.864701 19275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:02.865646 19275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:02.867081 19275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:02.867185 19275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:02.867333 19275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:02.867400 19275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:02.902218 19275 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:38375
I20260812 11:24:02.902266 19481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:38375 every 8 connection(s)
I20260812 11:24:02.903551 19275 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:02.904143 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19275
I20260812 11:24:02.904367 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:24:02.905020 12731 external_mini_cluster.cc:1428] Setting key dd17efaead2962d96fc620f1443292b3
Aug 12 11:24:02 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:02.926785 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:02.906280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:57467 (local address 127.12.110.254:41995)
0812 11:24:02.906862 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:02.906883 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:02.909930 (+ 3047us) server_negotiation.cc:407] Connection header received
0812 11:24:02.909985 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:02.909988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:02.910064 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:02.910150 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:02.911779 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.912610 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.914776 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.915446 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.915585 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:02.920207 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:02.920223 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:02.920229 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:02.920290 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:02.923425 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:02.924160 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:02.924164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:02.924167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:02.924304 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:02.924818 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:02.924823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:02.924826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:02.925245 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:02.925424 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:02.926228 (+ 804us) server_negotiation.cc:299] Negotiation successful
0812 11:24:02.926418 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":372,"threads_started":1}
I20260812 11:24:02.927834 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:02.907052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41995 (local address 127.12.110.195:57467)
0812 11:24:02.907905 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:02.907963 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:02.909675 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:02.910434 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:02.910442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:02.911049 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:02.911607 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:02.911625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.912794 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.912800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:02.914609 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:02.914627 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.915934 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.915949 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:02.916326 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:02.917191 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:02.917224 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:02.919820 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:02.923567 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:02.923575 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:02.923588 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:02.924025 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:02.924416 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:02.924420 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:02.924422 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:02.924609 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:02.925418 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:02.925425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:02.925794 (+ 369us) client_negotiation.cc:769] Sending connection context
0812 11:24:02.926064 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 11:24:02.926470 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":357,"threads_started":1}
I20260812 11:24:02.929224 19482 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41995
I20260812 11:24:02.929589 19482 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:02.930406 19482 heartbeater.cc:511] Master 127.12.110.254:41995 requested a full tablet report, sending...
I20260812 11:24:02.931978 18770 ts_manager.cc:194] Registered new tserver with Master: c70eb5c6a0134d0cad8ef4a31a561faa (127.12.110.195:38375)
I20260812 11:24:02.933687 18770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:57467
I20260812 11:24:02.934767 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:24:02 dist-test-slave-h6n0 krb5kdc[18694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:24:02 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:24:02.993260 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:02.977575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.12.110.193:35345)
0812 11:24:02.978269 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:02.978278 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:02.978312 (+ 34us) server_negotiation.cc:407] Connection header received
0812 11:24:02.978462 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:02.978470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:02.978712 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:02.978887 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:02.980327 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.981337 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.982573 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:02.983121 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:02.983259 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:02.987299 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:02.987311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:02.987324 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:02.987364 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:02.990406 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:02.991096 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:02.991103 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:02.991105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:02.991247 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:02.991622 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:02.991628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:02.991632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:02.991993 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:02.992304 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:02.992618 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 11:24:02.992893 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
Aug 12 11:24:03 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:03.013221 19484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:02.998131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.12.110.254:41995)
0812 11:24:02.998316 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:02.998321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:02.998761 (+ 440us) server_negotiation.cc:407] Connection header received
0812 11:24:02.998914 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:02.998919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:02.999018 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:02.999119 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:03.000624 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.001361 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.002982 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.003387 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.003481 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:03.007735 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:03.007741 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:03.007744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:03.007780 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:03.010639 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.011312 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.011316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.011319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.011443 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.011836 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.011840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.011842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.012182 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:03.012389 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:03.012690 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 11:24:03.012859 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 11:24:03.014048 18769 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50230
I20260812 11:24:03.015677 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18873
I20260812 11:24:03.030966 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:03.016787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.12.110.193:35345)
0812 11:24:03.017011 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:03.017017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:03.017125 (+ 108us) server_negotiation.cc:407] Connection header received
0812 11:24:03.017309 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:03.017315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:03.017392 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:03.017494 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:03.019186 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.020142 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.021835 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.022348 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.022515 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:03.025281 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:03.025287 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:03.025290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:03.025330 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:03.028363 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.029006 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.029012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.029016 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.029147 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.029528 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.029533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.029537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.029805 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:03.030065 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:03.030383 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 11:24:03.030639 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:24:03.031626 19061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:03.115262 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18873
I20260812 11:24:03.141521 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19077
Aug 12 11:24:03 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:24:03.160238 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:03.142809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.12.110.194:45879)
0812 11:24:03.143432 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:03.143439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:03.143464 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:24:03.143525 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:03.143532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:03.143762 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:03.143933 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:03.145653 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.147156 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.148624 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.149198 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.149356 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:03.154066 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:03.154078 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:03.154113 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:03.154165 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:03.157274 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.157968 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.157975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.157978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.158110 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.158498 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.158503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.158506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.158926 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:03.159266 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:03.159570 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 11:24:03.159875 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":306,"threads_started":1}
I20260812 11:24:03.161487 19259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:03.244905 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19077
I20260812 11:24:03.270846 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19275
Aug 12 11:24:03 dist-test-slave-h6n0 krb5kdc[18694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:24:03.288926 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:03.271920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58188 (local address 127.12.110.195:38375)
0812 11:24:03.272619 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:03.272627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:03.272660 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:24:03.272734 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:03.272741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:03.272965 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:03.273115 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:03.274426 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.275926 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.277240 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:03.277849 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:03.277956 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:03.282766 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:03.282778 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:03.282793 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:03.282829 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:03.285870 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.286581 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.286595 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.286597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.286759 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:03.287150 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:03.287154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:03.287157 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:03.287540 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:03.287803 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:03.288405 (+ 602us) server_negotiation.cc:299] Negotiation successful
0812 11:24:03.288620 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":361,"threads_started":1}
I20260812 11:24:03.290237 19474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:03.374696 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19275
I20260812 11:24:03.401365 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18714
I20260812 11:24:03.402560 18823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:03.514503 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18714
2026-08-12T11:24:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4515 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:24:03 dist-test-slave-h6n0 krb5kdc[19517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:24:03 dist-test-slave-h6n0 krb5kdc[19517](info): set up 2 sockets
Aug 12 11:24:03 dist-test-slave-h6n0 krb5kdc[19517](info): commencing operation
krb5kdc: starting...
W20260812 11:24:04.891264 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.323s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:24:05 dist-test-slave-h6n0 krb5kdc[19517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:24:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:24:05.155818 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:44671
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:44671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:05.504499 19537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:05.504894 19537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:05.505076 19537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:05.513139 19537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:05.513298 19537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:05.513412 19537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:05.513522 19537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:05.513612 19537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:05.524086 19537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:44671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:05.525430 19537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:05.526938 19537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:05.540143 19545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:05.541096 19547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:05.541244 19544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:05.541440 19537 server_base.cc:1034] running on GCE node
I20260812 11:24:05.543021 19537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:05.545181 19537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:05.548604 19537 hybrid_clock.cc:648] HybridClock initialized: now 1786533845548454 us; error 130 us; skew 500 ppm
I20260812 11:24:05.549405 19537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:05 dist-test-slave-h6n0 krb5kdc[19517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533845, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:05.553449 19537 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:24:05.554834 19537 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:05.554955 19537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:05.555317 19537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:05.558400 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "787f70d87839413aa81edd58df0385e8"
format_stamp: "Formatted at 2026-08-12 11:24:05 on dist-test-slave-h6n0"
server_key: "4b8bcbb6e8a1a66d9cf6e5b84db3a749"
server_key_iv: "10363e417abd4d4734de2317aa134bd7"
server_key_version: "encryptionkey@0"
I20260812 11:24:05.559322 19537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "787f70d87839413aa81edd58df0385e8"
format_stamp: "Formatted at 2026-08-12 11:24:05 on dist-test-slave-h6n0"
server_key: "4b8bcbb6e8a1a66d9cf6e5b84db3a749"
server_key_iv: "10363e417abd4d4734de2317aa134bd7"
server_key_version: "encryptionkey@0"
I20260812 11:24:05.564471 19537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260812 11:24:05.568416 19557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:05.569746 19537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:24:05.569962 19537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "787f70d87839413aa81edd58df0385e8"
format_stamp: "Formatted at 2026-08-12 11:24:05 on dist-test-slave-h6n0"
server_key: "4b8bcbb6e8a1a66d9cf6e5b84db3a749"
server_key_iv: "10363e417abd4d4734de2317aa134bd7"
server_key_version: "encryptionkey@0"
I20260812 11:24:05.570197 19537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:05.594728 19537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:05.600454 19537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:05.600836 19537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:05.624729 19649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:44671 every 8 connection(s)
I20260812 11:24:05.624766 19537 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:44671
I20260812 11:24:05.626108 19537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:05.630302 19652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:05.631635 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19537
I20260812 11:24:05.631909 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:05.632777 12731 external_mini_cluster.cc:1428] Setting key 61a1e19cc28b8c47b6dccf9267998d63
I20260812 11:24:05.644469 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8: Bootstrap starting.
I20260812 11:24:05.648514 19652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8: Neither blocks nor log segments found. Creating new log.
Aug 12 11:24:05 dist-test-slave-h6n0 krb5kdc[19517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:05.649715 19652 log.cc:824] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:05.653401 19652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8: No bootstrap required, opened a new log
I20260812 11:24:05.657929 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:05.635028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.12.110.254:44671)
0812 11:24:05.635889 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:05.635900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:05.635941 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:24:05.637609 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:05.637631 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:05.638004 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:05.638299 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:05.641303 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:05.642816 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:05.644750 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:05.645478 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:05.645608 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:05.650433 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:05.650441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:05.650455 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:05.650495 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:05.653803 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:05.654481 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:05.654488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:05.654493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:05.654727 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:05.655138 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:05.655142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:05.655144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:05.655663 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:05.656022 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:05.656470 (+ 448us) server_negotiation.cc:299] Negotiation successful
0812 11:24:05.656925 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":437,"threads_started":1}
I20260812 11:24:05.663949 19652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787f70d87839413aa81edd58df0385e8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44671 } }
I20260812 11:24:05.664378 19652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:05.664499 19652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787f70d87839413aa81edd58df0385e8, State: Initialized, Role: FOLLOWER
I20260812 11:24:05.665179 19652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787f70d87839413aa81edd58df0385e8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44671 } }
I20260812 11:24:05.665477 19652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:05.665643 19652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:05.665839 19652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:05.668347 19652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787f70d87839413aa81edd58df0385e8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44671 } }
I20260812 11:24:05.668886 19652 leader_election.cc:302] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787f70d87839413aa81edd58df0385e8; no voters:
I20260812 11:24:05.669497 19652 leader_election.cc:288] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:05.669652 19657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:05.670463 19657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [term 1 LEADER]: Becoming Leader. State: Replica: 787f70d87839413aa81edd58df0385e8, State: Running, Role: LEADER
I20260812 11:24:05.671078 19657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787f70d87839413aa81edd58df0385e8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44671 } }
I20260812 11:24:05.671931 19652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:05.673687 19661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787f70d87839413aa81edd58df0385e8. Latest consensus state: current_term: 1 leader_uuid: "787f70d87839413aa81edd58df0385e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787f70d87839413aa81edd58df0385e8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44671 } } }
I20260812 11:24:05.674072 19661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:05.673851 19658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787f70d87839413aa81edd58df0385e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787f70d87839413aa81edd58df0385e8" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44671 } } }
I20260812 11:24:05.674817 19658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:05.682348 19664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:05.686388 19664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:05.696118 19664 catalog_manager.cc:1347] Generated new cluster ID: 05ff588c4aeb44348eeeb94e377b9ed7
I20260812 11:24:05.696261 19664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:05.719852 19664 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:05.721769 19664 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:05.734982 19664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8: Generated new TSK 0
I20260812 11:24:05.736008 19664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:24:05.802677 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44671
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:06.150200 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:06.150484 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:06.150647 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:06.160912 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:06.161031 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:06.161159 19701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:06.171353 19701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:06.172711 19701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:06.174104 19701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:06.187978 19716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:06.188110 19714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:06.188110 19710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:06.192243 19701 server_base.cc:1034] running on GCE node
I20260812 11:24:06.192847 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:06.193779 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:06.195082 19701 hybrid_clock.cc:648] HybridClock initialized: now 1786533846195026 us; error 66 us; skew 500 ppm
I20260812 11:24:06.195438 19701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:06 dist-test-slave-h6n0 krb5kdc[19517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533846, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:06.202910 19701 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:24:06.204443 19701 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:06.204577 19701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:06.205009 19701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:06.208312 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "1f2feaa3cb2e46868965b759fe44dde4"
format_stamp: "Formatted at 2026-08-12 11:24:06 on dist-test-slave-h6n0"
server_key: "a0e75eee2c0f5a0b861b20b8fbde5932"
server_key_iv: "0760ac525db3d82275cb82a2cc5a0a58"
server_key_version: "encryptionkey@0"
I20260812 11:24:06.209141 19701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "1f2feaa3cb2e46868965b759fe44dde4"
format_stamp: "Formatted at 2026-08-12 11:24:06 on dist-test-slave-h6n0"
server_key: "a0e75eee2c0f5a0b861b20b8fbde5932"
server_key_iv: "0760ac525db3d82275cb82a2cc5a0a58"
server_key_version: "encryptionkey@0"
I20260812 11:24:06.214285 19701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:24:06.218052 19725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:06.219460 19701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:24:06.219652 19701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "1f2feaa3cb2e46868965b759fe44dde4"
format_stamp: "Formatted at 2026-08-12 11:24:06 on dist-test-slave-h6n0"
server_key: "a0e75eee2c0f5a0b861b20b8fbde5932"
server_key_iv: "0760ac525db3d82275cb82a2cc5a0a58"
server_key_version: "encryptionkey@0"
I20260812 11:24:06.219882 19701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:06.238854 19701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:06.242998 19701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:06.243278 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:06.244235 19701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:06.245653 19701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:06.245774 19701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:06.245911 19701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:06.245996 19701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:06.286404 19701 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:36697
I20260812 11:24:06.286455 19921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:36697 every 8 connection(s)
I20260812 11:24:06.287312 19701 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:06.289203 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19701
I20260812 11:24:06.289453 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:24:06.290217 12731 external_mini_cluster.cc:1428] Setting key 8acd74c406257021ac310a92d1f47318
Aug 12 11:24:06 dist-test-slave-h6n0 krb5kdc[19517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533846, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:06.313498 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:06.290121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:36587 (local address 127.12.110.254:44671)
0812 11:24:06.292568 (+ 2447us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:06.292572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:06.292589 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:24:06.294567 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:06.294575 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:06.294676 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:06.294771 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:06.296828 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.297564 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.300093 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.300568 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.300676 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:06.305555 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:06.305564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:06.305572 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:06.305618 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:06.309529 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:06.310456 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:06.310472 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:06.310478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:06.310672 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:06.311216 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:06.311224 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:06.311229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:06.311654 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:06.311805 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:06.312685 (+ 880us) server_negotiation.cc:299] Negotiation successful
0812 11:24:06.312990 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2340,"thread_start_us":2234,"threads_started":1}
I20260812 11:24:06.315389 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:06.290776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44671 (local address 127.12.110.193:36587)
0812 11:24:06.291723 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:06.291773 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:06.294236 (+ 2463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:06.295042 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:06.295056 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:06.295857 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:06.296642 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:06.296669 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.297723 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.297731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:06.299919 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:06.299937 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.300802 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.300812 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:06.301144 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:06.301937 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:06.301971 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:06.305234 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:06.309744 (+ 4510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:06.309752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:06.309766 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:06.310301 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:06.310816 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:06.310821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:06.310824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:06.311070 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:06.311823 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:06.311830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:06.312215 (+ 385us) client_negotiation.cc:769] Sending connection context
0812 11:24:06.312468 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 11:24:06.313305 (+ 837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":403,"threads_started":1}
I20260812 11:24:06.317287 19923 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44671
I20260812 11:24:06.317656 19923 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:06.318351 19923 heartbeater.cc:511] Master 127.12.110.254:44671 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
I20260812 11:24:06.320576 19593 ts_manager.cc:194] Registered new tserver with Master: 1f2feaa3cb2e46868965b759fe44dde4 (127.12.110.193:36697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
I20260812 11:24:06.322248 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:36587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:24:06.351905 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44671
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:06.703514 19931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:06.703878 19931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:06.704061 19931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:06.711700 19931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:06.711869 19931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:06.712077 19931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:06.722991 19931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:06.724458 19931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:06.725845 19931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:06.739506 19937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:06.739529 19939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:06.740121 19944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:06.741123 19931 server_base.cc:1034] running on GCE node
I20260812 11:24:06.741758 19931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:06.742758 19931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:06.743978 19931 hybrid_clock.cc:648] HybridClock initialized: now 1786533846743930 us; error 72 us; skew 500 ppm
I20260812 11:24:06.744297 19931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:06 dist-test-slave-h6n0 krb5kdc[19517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533846, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:06.747893 19931 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:24:06.749130 19931 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:06.749225 19931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:06.749497 19931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:06.752581 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "cb8617edf5024850bf47520850897d44"
format_stamp: "Formatted at 2026-08-12 11:24:06 on dist-test-slave-h6n0"
server_key: "6da45a09b13fd33e26cfcc4168e39e9d"
server_key_iv: "fe298ddf3689ec5dab50e980e7483956"
server_key_version: "encryptionkey@0"
I20260812 11:24:06.753365 19931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "cb8617edf5024850bf47520850897d44"
format_stamp: "Formatted at 2026-08-12 11:24:06 on dist-test-slave-h6n0"
server_key: "6da45a09b13fd33e26cfcc4168e39e9d"
server_key_iv: "fe298ddf3689ec5dab50e980e7483956"
server_key_version: "encryptionkey@0"
I20260812 11:24:06.760228 19931 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260812 11:24:06.764122 19951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:06.766094 19931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:24:06.766259 19931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "cb8617edf5024850bf47520850897d44"
format_stamp: "Formatted at 2026-08-12 11:24:06 on dist-test-slave-h6n0"
server_key: "6da45a09b13fd33e26cfcc4168e39e9d"
server_key_iv: "fe298ddf3689ec5dab50e980e7483956"
server_key_version: "encryptionkey@0"
I20260812 11:24:06.766464 19931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:06.795725 19931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:06.800830 19931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:06.801103 19931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:06.802106 19931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:06.803726 19931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:06.803840 19931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:06.803926 19931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:06.803992 19931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:06.844808 19931 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:34797
I20260812 11:24:06.844929 20134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:34797 every 8 connection(s)
I20260812 11:24:06.845764 19931 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:06.849368 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19931
I20260812 11:24:06.849606 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:24:06.850210 12731 external_mini_cluster.cc:1428] Setting key 478e70239b15f9140ce5e66b42c9b4b7
Aug 12 11:24:06 dist-test-slave-h6n0 krb5kdc[19517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533846, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:06.869818 20138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:06.849447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:45185 (local address 127.12.110.254:44671)
0812 11:24:06.850179 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:06.850187 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:06.851942 (+ 1755us) server_negotiation.cc:407] Connection header received
0812 11:24:06.854020 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:06.854025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:06.854148 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:06.854272 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:06.856021 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.856799 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.858487 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.859013 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.859155 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:06.862967 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:06.862972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:06.862975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:06.863010 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:06.865816 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:06.866882 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:06.866887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:06.866892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:06.867087 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:06.867598 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:06.867606 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:06.867612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:06.868023 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:06.868193 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:06.869043 (+ 850us) server_negotiation.cc:299] Negotiation successful
0812 11:24:06.869306 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":426,"threads_started":1}
I20260812 11:24:06.870797 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:06.850429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44671 (local address 127.12.110.194:45185)
0812 11:24:06.851607 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:06.851679 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:06.853720 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:06.854586 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:06.854601 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:06.855187 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:06.855811 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:06.855830 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.856984 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.856988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:06.858306 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:06.858318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:06.859282 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:06.859290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:06.859522 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:06.860297 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:06.860322 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:06.862688 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:06.865995 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:06.866003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:06.866017 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:06.866734 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:06.867206 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:06.867209 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:06.867212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:06.867466 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:06.868215 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:06.868224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:06.868631 (+ 407us) client_negotiation.cc:769] Sending connection context
0812 11:24:06.868897 (+ 266us) client_negotiation.cc:240] Negotiation successful
0812 11:24:06.869312 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":493,"threads_started":1}
I20260812 11:24:06.872653 20136 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44671
I20260812 11:24:06.873303 20136 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:06.874155 20136 heartbeater.cc:511] Master 127.12.110.254:44671 requested a full tablet report, sending...
I20260812 11:24:06.876039 19593 ts_manager.cc:194] Registered new tserver with Master: cb8617edf5024850bf47520850897d44 (127.12.110.194:34797)
I20260812 11:24:06.877367 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:45185
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:24:06.915160 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44671
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:07.266062 20145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:07.266328 20145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:07.266505 20145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:07.274057 20145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:07.274163 20145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:07.274359 20145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:07.283903 20145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:07.285199 20145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:07.286572 20145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:07.300551 20158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:07.300696 20159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:07.301131 20164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:07.304458 20145 server_base.cc:1034] running on GCE node
I20260812 11:24:07.305446 20145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:07.306947 20145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:07.308290 20145 hybrid_clock.cc:648] HybridClock initialized: now 1786533847308242 us; error 45 us; skew 500 ppm
I20260812 11:24:07.308928 20145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:07 dist-test-slave-h6n0 krb5kdc[19517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533847, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:07.313508 20145 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:24:07.315120 20145 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:07.315258 20145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:07.316012 20145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:07.319988 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "5fe2e02b54e84eabb3bca60527e0284d"
format_stamp: "Formatted at 2026-08-12 11:24:07 on dist-test-slave-h6n0"
server_key: "189ec07cd951bdcc9cbeae6c53100a49"
server_key_iv: "829ee205ddcf4e1165c9739cb7808d04"
server_key_version: "encryptionkey@0"
I20260812 11:24:07.320950 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "5fe2e02b54e84eabb3bca60527e0284d"
format_stamp: "Formatted at 2026-08-12 11:24:07 on dist-test-slave-h6n0"
server_key: "189ec07cd951bdcc9cbeae6c53100a49"
server_key_iv: "829ee205ddcf4e1165c9739cb7808d04"
server_key_version: "encryptionkey@0"
I20260812 11:24:07.326730 20145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:24:07.326987 19923 heartbeater.cc:503] Master 127.12.110.254:44671 was elected leader, sending a full tablet report...
I20260812 11:24:07.330755 20173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:07.332334 20145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:24:07.332554 20145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "5fe2e02b54e84eabb3bca60527e0284d"
format_stamp: "Formatted at 2026-08-12 11:24:07 on dist-test-slave-h6n0"
server_key: "189ec07cd951bdcc9cbeae6c53100a49"
server_key_iv: "829ee205ddcf4e1165c9739cb7808d04"
server_key_version: "encryptionkey@0"
I20260812 11:24:07.332790 20145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:07.346781 20145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:07.352064 20145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:07.352377 20145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:07.353394 20145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:07.354900 20145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:07.355027 20145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:07.355165 20145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:07.355260 20145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:07.392618 20145 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:42273
I20260812 11:24:07.392710 20367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:42273 every 8 connection(s)
I20260812 11:24:07.393523 20145 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:07.397315 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 20145
I20260812 11:24:07.397521 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:24:07.397984 12731 external_mini_cluster.cc:1428] Setting key 32b4ea56f37b97e6b6948446793a2063
Aug 12 11:24:07 dist-test-slave-h6n0 krb5kdc[19517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533847, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:07.414328 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.396016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:44517 (local address 127.12.110.254:44671)
0812 11:24:07.396523 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.396527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.397351 (+ 824us) server_negotiation.cc:407] Connection header received
0812 11:24:07.399838 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.399844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.399981 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.400100 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:07.401906 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.402733 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.404239 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.404624 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.404713 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.408045 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:07.408049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:07.408053 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:07.408098 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:07.410836 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.411556 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.411566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.411568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.411719 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.412296 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.412302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.412307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.412653 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:07.412816 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.413855 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.414050 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260812 11:24:07.415126 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.396471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44671 (local address 127.12.110.195:44517)
0812 11:24:07.397201 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.397243 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.399535 (+ 2292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.400392 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.400400 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.401042 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:07.401710 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.401728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.402870 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.402874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.404119 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.404131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.404808 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.404812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.404983 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.405604 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:07.405631 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:07.407826 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:07.410975 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:07.410984 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:07.411000 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:07.411419 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:07.411873 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:07.411877 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:07.411880 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:07.412134 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:07.412818 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:07.412826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:07.413256 (+ 430us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.413551 (+ 295us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.413998 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":316,"threads_started":1}
I20260812 11:24:07.416381 20370 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44671
I20260812 11:24:07.416699 20370 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:07.417369 20370 heartbeater.cc:511] Master 127.12.110.254:44671 requested a full tablet report, sending...
I20260812 11:24:07.418795 19593 ts_manager.cc:194] Registered new tserver with Master: 5fe2e02b54e84eabb3bca60527e0284d (127.12.110.195:42273)
I20260812 11:24:07.419600 19593 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:44517
I20260812 11:24:07.426018 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:24:07.443267 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.430747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.12.110.254:44671)
0812 11:24:07.430967 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.430971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.431497 (+ 526us) server_negotiation.cc:407] Connection header received
0812 11:24:07.431641 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.431645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.431706 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.431786 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:07.433115 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.433814 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.435108 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.435535 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.435641 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.437791 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:07.437797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:07.437801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:07.437843 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:07.440415 (+ 2572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.441166 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.441173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.441176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.441318 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.441723 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.441728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.441732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.442083 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:07.442362 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.442739 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.442968 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:24:07.448029 19593 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:07.455169 19593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:07.492194 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.468365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.12.110.195:42273)
0812 11:24:07.481000 (+ 12635us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.481024 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.481073 (+ 49us) server_negotiation.cc:407] Connection header received
0812 11:24:07.481210 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.481221 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.481567 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.481842 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:07.483443 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.485614 (+ 2171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.488792 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.490436 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.491376 (+ 940us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.491583 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.491686 (+ 103us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.491807 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12373,"thread_start_us":13757,"threads_started":1}
I20260812 11:24:07.492652 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.468156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:42273 (local address 127.0.0.1:55874)
0812 11:24:07.480482 (+ 12326us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.480561 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.480810 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.482464 (+ 1654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.482475 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.482518 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:07.483118 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.483136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.485841 (+ 2705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.485850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.488615 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.488633 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.490900 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.490910 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.491273 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.491351 (+ 78us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.491460 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.492174 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11950,"thread_start_us":15291,"threads_started":1}
I20260812 11:24:07.496704 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.468589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.12.110.194:34797)
0812 11:24:07.487585 (+ 18996us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.487595 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.487628 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:24:07.487770 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.487783 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.488093 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.488301 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:07.489478 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.490933 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.494677 (+ 3744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.495621 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.495827 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.495998 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.496204 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.496374 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18829,"thread_start_us":19064,"threads_started":1}
I20260812 11:24:07.496791 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.468259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:34797 (local address 127.0.0.1:54522)
0812 11:24:07.483171 (+ 14912us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.483200 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.483357 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.488615 (+ 5258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.488621 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.488645 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:07.489206 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.489219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.491216 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.491222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.494434 (+ 3212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.494454 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.495835 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.495841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.496001 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.496040 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.496161 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.496303 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14729,"mutex_wait_us":2,"spinlock_wait_cycles":10624,"thread_start_us":12824,"threads_started":1}
I20260812 11:24:07.497419 20399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.483840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36697 (local address 127.0.0.1:59720)
0812 11:24:07.487412 (+ 3572us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.487437 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.487576 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.488670 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.488677 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.488711 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:07.489211 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.489220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.490565 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.490573 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.493719 (+ 3146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.493736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.496471 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.496479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.496699 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.496740 (+ 41us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.496847 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.497004 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3422,"mutex_wait_us":1,"thread_start_us":3505,"threads_started":1}
I20260812 11:24:07.497589 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.482500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.12.110.193:36697)
0812 11:24:07.487277 (+ 4777us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.487290 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.487871 (+ 581us) server_negotiation.cc:407] Connection header received
0812 11:24:07.487968 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.487975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.488231 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.488402 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:07.489404 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.490373 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.493965 (+ 3592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.495900 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.496114 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.496273 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.497045 (+ 772us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.497242 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4583,"thread_start_us":4376,"threads_started":1}
I20260812 11:24:07.497856 20262 tablet_service.cc:1467] Processing CreateTablet for tablet b052de104b7f4d7c8f1855d9b1a80092 (DEFAULT_TABLE table=test-table [id=928539edddf34845b833a058c5c1f7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:07.500622 20262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b052de104b7f4d7c8f1855d9b1a80092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:07.500598 20036 tablet_service.cc:1467] Processing CreateTablet for tablet b052de104b7f4d7c8f1855d9b1a80092 (DEFAULT_TABLE table=test-table [id=928539edddf34845b833a058c5c1f7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:07.501166 19806 tablet_service.cc:1467] Processing CreateTablet for tablet b052de104b7f4d7c8f1855d9b1a80092 (DEFAULT_TABLE table=test-table [id=928539edddf34845b833a058c5c1f7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:07.501852 20036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b052de104b7f4d7c8f1855d9b1a80092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:07.502472 19806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b052de104b7f4d7c8f1855d9b1a80092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:07.513774 20400 tablet_bootstrap.cc:492] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Bootstrap starting.
I20260812 11:24:07.516856 20404 tablet_bootstrap.cc:492] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Bootstrap starting.
I20260812 11:24:07.518770 20400 tablet_bootstrap.cc:654] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:07.520159 20400 log.cc:824] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:07.520360 20404 tablet_bootstrap.cc:654] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:07.521181 20401 tablet_bootstrap.cc:492] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Bootstrap starting.
I20260812 11:24:07.522398 20404 log.cc:824] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:07.524839 20401 tablet_bootstrap.cc:654] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:07.525677 20404 tablet_bootstrap.cc:492] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: No bootstrap required, opened a new log
I20260812 11:24:07.525918 20400 tablet_bootstrap.cc:492] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: No bootstrap required, opened a new log
I20260812 11:24:07.525945 20404 ts_tablet_manager.cc:1397] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260812 11:24:07.526036 20401 log.cc:824] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:07.526196 20400 ts_tablet_manager.cc:1397] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260812 11:24:07.529112 20401 tablet_bootstrap.cc:492] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: No bootstrap required, opened a new log
I20260812 11:24:07.529351 20401 ts_tablet_manager.cc:1397] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260812 11:24:07.534610 20404 raft_consensus.cc:348] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.535312 20404 raft_consensus.cc:374] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:07.535430 20404 raft_consensus.cc:729] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f2feaa3cb2e46868965b759fe44dde4, State: Initialized, Role: FOLLOWER
I20260812 11:24:07.536091 20404 consensus_queue.cc:260] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.536792 20400 raft_consensus.cc:348] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.537238 20400 raft_consensus.cc:374] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:07.537374 20400 raft_consensus.cc:729] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb8617edf5024850bf47520850897d44, State: Initialized, Role: FOLLOWER
I20260812 11:24:07.538087 20404 ts_tablet_manager.cc:1428] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260812 11:24:07.538075 20400 consensus_queue.cc:260] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
W20260812 11:24:07.540078 19924 tablet.cc:2369] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:07.540411 20401 raft_consensus.cc:348] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.540975 20401 raft_consensus.cc:374] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:07.541329 20401 raft_consensus.cc:729] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fe2e02b54e84eabb3bca60527e0284d, State: Initialized, Role: FOLLOWER
I20260812 11:24:07.542115 20401 consensus_queue.cc:260] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.543198 20136 heartbeater.cc:503] Master 127.12.110.254:44671 was elected leader, sending a full tablet report...
I20260812 11:24:07.543922 20400 ts_tablet_manager.cc:1428] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Time spent starting tablet: real 0.018s user 0.017s sys 0.001s
I20260812 11:24:07.544646 20370 heartbeater.cc:503] Master 127.12.110.254:44671 was elected leader, sending a full tablet report...
I20260812 11:24:07.545310 20401 ts_tablet_manager.cc:1428] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260812 11:24:07.564502 20408 raft_consensus.cc:482] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:07.564973 20408 raft_consensus.cc:504] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.566905 20408 leader_election.cc:288] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fe2e02b54e84eabb3bca60527e0284d (127.12.110.195:42273), cb8617edf5024850bf47520850897d44 (127.12.110.194:34797)
I20260812 11:24:07.569190 20411 raft_consensus.cc:482] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:07.569536 20411 raft_consensus.cc:504] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.571717 20411 leader_election.cc:288] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5fe2e02b54e84eabb3bca60527e0284d (127.12.110.195:42273), 1f2feaa3cb2e46868965b759fe44dde4 (127.12.110.193:36697)
I20260812 11:24:07.578447 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.567221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:51235 (local address 127.12.110.195:42273)
0812 11:24:07.567479 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.567488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.568253 (+ 765us) server_negotiation.cc:407] Connection header received
0812 11:24:07.568484 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.568488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.568551 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.568658 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:07.569805 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.571103 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.574775 (+ 3672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.576694 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.576895 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.577014 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.577814 (+ 800us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.578040 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 11:24:07.579828 20416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.567208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:42273 (local address 127.12.110.193:51235)
0812 11:24:07.568111 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.568131 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.568308 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.568694 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.568699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.568722 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:07.569440 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.569478 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.571341 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.571347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.574519 (+ 3172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.574561 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.577284 (+ 2723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.577294 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.577558 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.577612 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.579030 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.579267 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":51,"thread_start_us":470,"threads_started":1}
I20260812 11:24:07.581354 20292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b052de104b7f4d7c8f1855d9b1a80092" candidate_uuid: "1f2feaa3cb2e46868965b759fe44dde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe2e02b54e84eabb3bca60527e0284d" is_pre_election: true
I20260812 11:24:07.581722 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.567678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:53489 (local address 127.12.110.194:34797)
0812 11:24:07.567894 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.567900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.568298 (+ 398us) server_negotiation.cc:407] Connection header received
0812 11:24:07.568476 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.568480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.568539 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.568606 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:07.569680 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.570905 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.574680 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.577325 (+ 2645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.577523 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.577642 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.581274 (+ 3632us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.581446 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:24:07.581895 20414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.567211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:34797 (local address 127.12.110.193:53489)
0812 11:24:07.568158 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.568177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.568337 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.568658 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.568662 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.568687 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:07.569449 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.569468 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.571086 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.571091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.574244 (+ 3153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.574269 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.580715 (+ 6446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.580725 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.581029 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.581082 (+ 53us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.581224 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.581431 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":243,"thread_start_us":599,"threads_started":1}
I20260812 11:24:07.582029 20292 raft_consensus.cc:2454] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f2feaa3cb2e46868965b759fe44dde4 in term 0.
I20260812 11:24:07.583034 19726 leader_election.cc:302] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f2feaa3cb2e46868965b759fe44dde4, 5fe2e02b54e84eabb3bca60527e0284d; no voters:
I20260812 11:24:07.583050 20065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b052de104b7f4d7c8f1855d9b1a80092" candidate_uuid: "1f2feaa3cb2e46868965b759fe44dde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8617edf5024850bf47520850897d44" is_pre_election: true
I20260812 11:24:07.583648 20408 raft_consensus.cc:2781] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:07.583663 20065 raft_consensus.cc:2454] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f2feaa3cb2e46868965b759fe44dde4 in term 0.
I20260812 11:24:07.583827 20408 raft_consensus.cc:482] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:07.583952 20408 raft_consensus.cc:3037] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:07.587821 20408 raft_consensus.cc:504] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.589146 20065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b052de104b7f4d7c8f1855d9b1a80092" candidate_uuid: "1f2feaa3cb2e46868965b759fe44dde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb8617edf5024850bf47520850897d44"
I20260812 11:24:07.589169 20292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b052de104b7f4d7c8f1855d9b1a80092" candidate_uuid: "1f2feaa3cb2e46868965b759fe44dde4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe2e02b54e84eabb3bca60527e0284d"
I20260812 11:24:07.589372 20292 raft_consensus.cc:3037] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:07.589370 20065 raft_consensus.cc:3037] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:07.592734 20292 raft_consensus.cc:2454] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f2feaa3cb2e46868965b759fe44dde4 in term 1.
I20260812 11:24:07.592731 20065 raft_consensus.cc:2454] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f2feaa3cb2e46868965b759fe44dde4 in term 1.
I20260812 11:24:07.593106 20408 leader_election.cc:288] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [CANDIDATE]: Term 1 election: Requested vote from peers 5fe2e02b54e84eabb3bca60527e0284d (127.12.110.195:42273), cb8617edf5024850bf47520850897d44 (127.12.110.194:34797)
I20260812 11:24:07.593169 19729 leader_election.cc:302] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f2feaa3cb2e46868965b759fe44dde4, cb8617edf5024850bf47520850897d44; no voters:
I20260812 11:24:07.593495 20408 raft_consensus.cc:2781] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:07.594447 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.573770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:39811 (local address 127.12.110.193:36697)
0812 11:24:07.574047 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.574059 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.587784 (+ 13725us) server_negotiation.cc:407] Connection header received
0812 11:24:07.588018 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.588030 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.588123 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.588209 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:07.589063 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.590104 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.592258 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.593290 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.593447 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.593532 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.594058 (+ 526us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.594189 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:24:07.594595 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.575112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36697 (local address 127.12.110.194:39811)
0812 11:24:07.587654 (+ 12542us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.587670 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.587833 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.588230 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.588235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.588260 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:07.588866 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.588879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.590385 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.590389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.592089 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.592102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.593689 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.593695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.593892 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.593917 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.594032 (+ 115us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.594176 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12423,"thread_start_us":12101,"threads_started":1}
I20260812 11:24:07.595533 19837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b052de104b7f4d7c8f1855d9b1a80092" candidate_uuid: "cb8617edf5024850bf47520850897d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f2feaa3cb2e46868965b759fe44dde4" is_pre_election: true
I20260812 11:24:07.597985 20408 raft_consensus.cc:686] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 1 LEADER]: Becoming Leader. State: Replica: 1f2feaa3cb2e46868965b759fe44dde4, State: Running, Role: LEADER
W20260812 11:24:07.598675 20137 tablet.cc:2369] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:07.598636 20408 consensus_queue.cc:237] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } }
I20260812 11:24:07.599807 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.572859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:54023 (local address 127.12.110.195:42273)
0812 11:24:07.578819 (+ 5960us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.578828 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.587200 (+ 8372us) server_negotiation.cc:407] Connection header received
0812 11:24:07.587513 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.587522 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.587610 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.587681 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:07.588634 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.589950 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.597706 (+ 7756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.598737 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.598913 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.598980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.599427 (+ 447us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.599561 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5813,"thread_start_us":13080,"threads_started":1}
I20260812 11:24:07.599893 20419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.572281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:42273 (local address 127.12.110.194:54023)
0812 11:24:07.587096 (+ 14815us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:07.587121 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:07.587315 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:07.587900 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:07.587905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:07.587942 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:07.588470 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.588483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.594783 (+ 6300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.594792 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.597524 (+ 2732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:07.597543 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.599065 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.599071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:07.599240 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.599272 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:24:07.599376 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 11:24:07.599507 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14673,"mutex_wait_us":60,"thread_start_us":14477,"threads_started":1}
I20260812 11:24:07.600627 20292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b052de104b7f4d7c8f1855d9b1a80092" candidate_uuid: "cb8617edf5024850bf47520850897d44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fe2e02b54e84eabb3bca60527e0284d" is_pre_election: true
I20260812 11:24:07.600998 20292 raft_consensus.cc:2379] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cb8617edf5024850bf47520850897d44 in current term 1: Already voted for candidate 1f2feaa3cb2e46868965b759fe44dde4 in this term.
I20260812 11:24:07.603875 19959 leader_election.cc:302] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cb8617edf5024850bf47520850897d44; no voters: 1f2feaa3cb2e46868965b759fe44dde4, 5fe2e02b54e84eabb3bca60527e0284d
I20260812 11:24:07.604007 19590 catalog_manager.cc:5495] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f2feaa3cb2e46868965b759fe44dde4 (127.12.110.193). New cstate: current_term: 1 leader_uuid: "1f2feaa3cb2e46868965b759fe44dde4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f2feaa3cb2e46868965b759fe44dde4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36697 } health_report { overall_health: HEALTHY } } }
I20260812 11:24:07.604637 20411 raft_consensus.cc:2726] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 11:24:07.634684 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.626743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.12.110.193:36697)
0812 11:24:07.626951 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.626959 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.627232 (+ 273us) server_negotiation.cc:407] Connection header received
0812 11:24:07.627350 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.627353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.627431 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.627602 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:07.627973 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.628854 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.630907 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.631522 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.631700 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.632480 (+ 780us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:07.633550 (+ 1070us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:07.633829 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.634054 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.634272 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:24:07.643455 20292 raft_consensus.cc:1261] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 1 FOLLOWER]: Refusing update from remote peer 1f2feaa3cb2e46868965b759fe44dde4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:07.643464 20065 raft_consensus.cc:1261] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 1 FOLLOWER]: Refusing update from remote peer 1f2feaa3cb2e46868965b759fe44dde4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:07.644346 20425 consensus_queue.cc:1035] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fe2e02b54e84eabb3bca60527e0284d" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 42273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:07.644747 20408 consensus_queue.cc:1035] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb8617edf5024850bf47520850897d44" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:24:07.645385 20371 tablet.cc:2369] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:07.656692 20429 mvcc.cc:205] Tried to move back new op lower bound from 7317642639937036288 to 7317642639768444928. Current Snapshot: MvccSnapshot[applied={T|T < 7317642639937036288}]
I20260812 11:24:07.661072 20431 mvcc.cc:205] Tried to move back new op lower bound from 7317642639937036288 to 7317642639768444928. Current Snapshot: MvccSnapshot[applied={T|T < 7317642639937036288}]
I20260812 11:24:07.674784 19590 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:07.675096 19590 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:07.678362 19590 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787f70d87839413aa81edd58df0385e8: Sending DeleteTablet for 3 replicas of tablet b052de104b7f4d7c8f1855d9b1a80092
I20260812 11:24:07.679579 20262 tablet_service.cc:1514] Processing DeleteTablet for tablet b052de104b7f4d7c8f1855d9b1a80092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:07 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:55874
I20260812 11:24:07.680272 20036 tablet_service.cc:1514] Processing DeleteTablet for tablet b052de104b7f4d7c8f1855d9b1a80092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:07 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:54522
I20260812 11:24:07.680917 19806 tablet_service.cc:1514] Processing DeleteTablet for tablet b052de104b7f4d7c8f1855d9b1a80092 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:07 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:59720
I20260812 11:24:07.681362 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19701
I20260812 11:24:07.689327 20445 tablet_replica.cc:330] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: stopping tablet replica
I20260812 11:24:07.689808 20445 raft_consensus.cc:2229] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:07.690557 20445 raft_consensus.cc:2258] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:07.692970 20445 ts_tablet_manager.cc:1905] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:07.696630 20443 tablet_replica.cc:330] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: stopping tablet replica
I20260812 11:24:07.697592 20443 raft_consensus.cc:2229] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:07.698146 20443 raft_consensus.cc:2258] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:07.699014 20444 tablet_replica.cc:330] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: stopping tablet replica
I20260812 11:24:07.699481 20444 raft_consensus.cc:2229] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:07.699949 20444 raft_consensus.cc:2258] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:07.700776 20445 ts_tablet_manager.cc:1918] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:07.700966 20445 log.cc:1196] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/b052de104b7f4d7c8f1855d9b1a80092
I20260812 11:24:07.701547 20445 ts_tablet_manager.cc:1939] T b052de104b7f4d7c8f1855d9b1a80092 P 1f2feaa3cb2e46868965b759fe44dde4: Deleting consensus metadata
I20260812 11:24:07.702872 20444 ts_tablet_manager.cc:1905] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:07.703797 19562 catalog_manager.cc:4841] TS 1f2feaa3cb2e46868965b759fe44dde4 (127.12.110.193:36697): tablet b052de104b7f4d7c8f1855d9b1a80092 (table test-table [id=928539edddf34845b833a058c5c1f7df]) successfully deleted
I20260812 11:24:07.705511 20443 ts_tablet_manager.cc:1905] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 11:24:07 dist-test-slave-h6n0 krb5kdc[19517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:24:07.710480 20444 ts_tablet_manager.cc:1918] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:07.710675 20444 log.cc:1196] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/b052de104b7f4d7c8f1855d9b1a80092
I20260812 11:24:07.711227 20444 ts_tablet_manager.cc:1939] T b052de104b7f4d7c8f1855d9b1a80092 P cb8617edf5024850bf47520850897d44: Deleting consensus metadata
I20260812 11:24:07.713104 19562 catalog_manager.cc:4841] TS cb8617edf5024850bf47520850897d44 (127.12.110.194:34797): tablet b052de104b7f4d7c8f1855d9b1a80092 (table test-table [id=928539edddf34845b833a058c5c1f7df]) successfully deleted
I20260812 11:24:07.713204 20443 ts_tablet_manager.cc:1918] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:07.713389 20443 log.cc:1196] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/b052de104b7f4d7c8f1855d9b1a80092
I20260812 11:24:07.713918 20443 ts_tablet_manager.cc:1939] T b052de104b7f4d7c8f1855d9b1a80092 P 5fe2e02b54e84eabb3bca60527e0284d: Deleting consensus metadata
I20260812 11:24:07.715638 20398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.682752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59730 (local address 127.12.110.193:36697)
0812 11:24:07.682950 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.682957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.689096 (+ 6139us) server_negotiation.cc:407] Connection header received
0812 11:24:07.690624 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.690635 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.690744 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.690841 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:07.696282 (+ 5441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.697853 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.700362 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.700889 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.701011 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.707724 (+ 6713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:07.707741 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:07.707763 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:07.707817 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:07.712167 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.713181 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.713189 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.713192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.713522 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.713921 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.713926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.713930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.714351 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:07.714718 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.715031 (+ 313us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.715285 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:24:07.715745 19560 catalog_manager.cc:4841] TS 5fe2e02b54e84eabb3bca60527e0284d (127.12.110.195:42273): tablet b052de104b7f4d7c8f1855d9b1a80092 (table test-table [id=928539edddf34845b833a058c5c1f7df]) successfully deleted
I20260812 11:24:07.716516 19911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:07.817351 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19701
I20260812 11:24:07.843704 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19931
Aug 12 11:24:07 dist-test-slave-h6n0 krb5kdc[19517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:24:07.862063 20396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.844842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.12.110.194:34797)
0812 11:24:07.845189 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.845195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.845217 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:24:07.845359 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.845363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.845429 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.845545 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:07.847224 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.848079 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.849891 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.850311 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.850418 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.855535 (+ 5117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:07.855546 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:07.855559 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:07.855590 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:07.858675 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.859456 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.859463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.859465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.859608 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.860012 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.860016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.860018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.860425 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:07.860663 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:07.861042 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 11:24:07.861596 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 11:24:07.862869 20130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:07.959764 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19931
I20260812 11:24:07.986073 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 20145
Aug 12 11:24:07 dist-test-slave-h6n0 krb5kdc[19517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:24:08.001832 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:07.987167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55876 (local address 127.12.110.195:42273)
0812 11:24:07.987336 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:07.987341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:07.987458 (+ 117us) server_negotiation.cc:407] Connection header received
0812 11:24:07.987620 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:07.987624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:07.987706 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:07.987796 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:07.989283 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.989974 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.991316 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:07.991718 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:07.991814 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:07.996115 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:07.996128 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:07.996143 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:07.996175 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:07.999116 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:07.999755 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:07.999762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:07.999765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:07.999893 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:08.000272 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:08.000276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:08.000278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:08.000684 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:08.000909 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:08.001211 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 11:24:08.001366 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:24:08.002574 20362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:08.095196 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 20145
I20260812 11:24:08.120600 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19537
I20260812 11:24:08.121935 19640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:08.238422 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19537
2026-08-12T11:24:08Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4730 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-12T11:24:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:08Z Disabled control of system clock
I20260812 11:24:08.345717 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42099
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:42099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 11:24:08.713120 20458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:08.713413 20458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:08.713557 20458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:08.720860 20458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:08.720973 20458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:08.721046 20458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:08.721166 20458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:08.721230 20458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:08.732573 20458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:42099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:08.733974 20458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:08.735841 20458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:08.750564 20465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:08.750699 20466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:08.750717 20468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:08.751888 20458 server_base.cc:1034] running on GCE node
I20260812 11:24:08.752789 20458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:08.754340 20458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:08.755587 20458 hybrid_clock.cc:648] HybridClock initialized: now 1786533848755520 us; error 65 us; skew 500 ppm
I20260812 11:24:08.756024 20458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:08.758472 20458 webserver.cc:460] Webserver started at http://127.12.110.254:43983/ using document root <none> and password file <none>
I20260812 11:24:08.759262 20458 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:08.759378 20458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:08.759730 20458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:08.762795 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "faa04bced8d14e7893ca131f43f8af14"
format_stamp: "Formatted at 2026-08-12 11:24:08 on dist-test-slave-h6n0"
server_key: "a1067ed84e829760aea05ba851054825"
server_key_iv: "28c410b9811e2ce17c5e60a39997125b"
server_key_version: "encryptionkey@0"
I20260812 11:24:08.763586 20458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "faa04bced8d14e7893ca131f43f8af14"
format_stamp: "Formatted at 2026-08-12 11:24:08 on dist-test-slave-h6n0"
server_key: "a1067ed84e829760aea05ba851054825"
server_key_iv: "28c410b9811e2ce17c5e60a39997125b"
server_key_version: "encryptionkey@0"
I20260812 11:24:08.768939 20458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:24:08.773701 20476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:08.775605 20458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260812 11:24:08.775849 20458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "faa04bced8d14e7893ca131f43f8af14"
format_stamp: "Formatted at 2026-08-12 11:24:08 on dist-test-slave-h6n0"
server_key: "a1067ed84e829760aea05ba851054825"
server_key_iv: "28c410b9811e2ce17c5e60a39997125b"
server_key_version: "encryptionkey@0"
I20260812 11:24:08.776130 20458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:08.782886 20458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:08.783746 20458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:08.784078 20458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:08.805759 20458 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:42099
I20260812 11:24:08.805781 20564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:42099 every 8 connection(s)
I20260812 11:24:08.807271 20458 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:08.811098 20566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:08.813201 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 20458
I20260812 11:24:08.813409 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:08.814307 12731 external_mini_cluster.cc:1428] Setting key 8b2c54f264a8bd4a848a71827b2f620f
I20260812 11:24:08.823112 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:08.816696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.12.110.254:42099)
0812 11:24:08.817740 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:08.817753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:08.817797 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:24:08.819494 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:08.819515 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:08.819884 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:08.820272 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:08.821352 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:08.821358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:08.821369 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:08.821383 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:08.821457 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:08.821475 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:08.821720 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:08.821872 (+ 152us) server_negotiation.cc:299] Negotiation successful
0812 11:24:08.822132 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":354,"threads_started":1}
I20260812 11:24:08.823920 20566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14: Bootstrap starting.
I20260812 11:24:08.827270 20566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:08.828382 20566 log.cc:824] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:08.831264 20566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14: No bootstrap required, opened a new log
I20260812 11:24:08.839206 20566 raft_consensus.cc:348] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa04bced8d14e7893ca131f43f8af14" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42099 } }
I20260812 11:24:08.839684 20566 raft_consensus.cc:374] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:08.839834 20566 raft_consensus.cc:729] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faa04bced8d14e7893ca131f43f8af14, State: Initialized, Role: FOLLOWER
I20260812 11:24:08.840516 20566 consensus_queue.cc:260] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa04bced8d14e7893ca131f43f8af14" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42099 } }
I20260812 11:24:08.840778 20566 raft_consensus.cc:388] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:08.840929 20566 raft_consensus.cc:482] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:08.841116 20566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:08.843441 20566 raft_consensus.cc:504] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa04bced8d14e7893ca131f43f8af14" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42099 } }
I20260812 11:24:08.843972 20566 leader_election.cc:302] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faa04bced8d14e7893ca131f43f8af14; no voters:
I20260812 11:24:08.844732 20566 leader_election.cc:288] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:08.844873 20575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:08.846050 20575 raft_consensus.cc:686] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [term 1 LEADER]: Becoming Leader. State: Replica: faa04bced8d14e7893ca131f43f8af14, State: Running, Role: LEADER
I20260812 11:24:08.846873 20575 consensus_queue.cc:237] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa04bced8d14e7893ca131f43f8af14" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42099 } }
I20260812 11:24:08.847581 20566 sys_catalog.cc:564] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:08.850236 20577 sys_catalog.cc:455] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faa04bced8d14e7893ca131f43f8af14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa04bced8d14e7893ca131f43f8af14" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42099 } } }
I20260812 11:24:08.850801 20577 sys_catalog.cc:458] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:08.851109 20578 sys_catalog.cc:455] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faa04bced8d14e7893ca131f43f8af14. Latest consensus state: current_term: 1 leader_uuid: "faa04bced8d14e7893ca131f43f8af14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faa04bced8d14e7893ca131f43f8af14" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42099 } } }
I20260812 11:24:08.851631 20578 sys_catalog.cc:458] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:08.852829 20591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:08.858721 20591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:08.870151 20591 catalog_manager.cc:1347] Generated new cluster ID: e725aa4935904c219de30d56df813cd9
I20260812 11:24:08.870328 20591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:08.881575 20591 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:08.881973 20591 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:08.894876 20591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14: Generated new TSK 0
I20260812 11:24:08.895944 20591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:24:08.913113 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42099
--builtin_ntp_servers=127.12.110.212:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260812 11:24:09.273317 20605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:09.273617 20605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:09.273772 20605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:09.281253 20605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:09.281354 20605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:09.281529 20605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:09.291167 20605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:09.292469 20605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:09.294385 20605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:09.307300 20613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:09.307340 20615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:09.307257 20618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:09.308311 20605 server_base.cc:1034] running on GCE node
I20260812 11:24:09.308977 20605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:09.309973 20605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:09.311272 20605 hybrid_clock.cc:648] HybridClock initialized: now 1786533849311233 us; error 55 us; skew 500 ppm
I20260812 11:24:09.311657 20605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:09.314280 20605 webserver.cc:460] Webserver started at http://127.12.110.193:35039/ using document root <none> and password file <none>
I20260812 11:24:09.315058 20605 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:09.315198 20605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:09.315558 20605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:09.318878 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "4150fd358f4d45418c016ca3d0044b10"
format_stamp: "Formatted at 2026-08-12 11:24:09 on dist-test-slave-h6n0"
server_key: "3c3dfde056a4dd34c9f36be0ab623fea"
server_key_iv: "fa63316f6107827e4c2b325f68b406be"
server_key_version: "encryptionkey@0"
I20260812 11:24:09.319722 20605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "4150fd358f4d45418c016ca3d0044b10"
format_stamp: "Formatted at 2026-08-12 11:24:09 on dist-test-slave-h6n0"
server_key: "3c3dfde056a4dd34c9f36be0ab623fea"
server_key_iv: "fa63316f6107827e4c2b325f68b406be"
server_key_version: "encryptionkey@0"
I20260812 11:24:09.325755 20605 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260812 11:24:09.329541 20629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:09.330928 20605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:24:09.331117 20605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "4150fd358f4d45418c016ca3d0044b10"
format_stamp: "Formatted at 2026-08-12 11:24:09 on dist-test-slave-h6n0"
server_key: "3c3dfde056a4dd34c9f36be0ab623fea"
server_key_iv: "fa63316f6107827e4c2b325f68b406be"
server_key_version: "encryptionkey@0"
I20260812 11:24:09.331355 20605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:09.337129 20605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:09.337993 20605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:09.338296 20605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:09.339356 20605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:09.340939 20605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:09.341096 20605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:09.341238 20605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:09.341331 20605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:09.378593 20605 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:43747
I20260812 11:24:09.378693 20829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:43747 every 8 connection(s)
I20260812 11:24:09.380128 20605 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:09.389590 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:09.383465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:58789 (local address 127.12.110.254:42099)
0812 11:24:09.384045 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:09.384053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:09.385303 (+ 1250us) server_negotiation.cc:407] Connection header received
0812 11:24:09.387177 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:09.387182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:09.387240 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:09.387341 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:09.388707 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:09.388711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:09.388714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:09.388729 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:09.388771 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:09.388787 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:09.388891 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:09.389192 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 11:24:09.389282 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260812 11:24:09.390570 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 20605
I20260812 11:24:09.390774 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:24:09.391063 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:09.384082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42099 (local address 127.12.110.193:58789)
0812 11:24:09.385120 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:09.385161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:09.386897 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:09.387609 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:09.387617 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:09.387701 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:09.388378 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:09.388381 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:09.388418 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:09.388499 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:09.388876 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:09.388884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:09.388890 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:24:09.389112 (+ 222us) client_negotiation.cc:240] Negotiation successful
0812 11:24:09.389497 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":314,"threads_started":1}
I20260812 11:24:09.391537 12731 external_mini_cluster.cc:1428] Setting key 1617d7ca7c8ef71ee3d941ca814815c0
I20260812 11:24:09.392985 20830 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42099
I20260812 11:24:09.393361 20830 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:09.394099 20830 heartbeater.cc:511] Master 127.12.110.254:42099 requested a full tablet report, sending...
I20260812 11:24:09.394851 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42099
--builtin_ntp_servers=127.12.110.212:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 11:24:09.396586 20500 ts_manager.cc:194] Registered new tserver with Master: 4150fd358f4d45418c016ca3d0044b10 (127.12.110.193:43747)
W20260812 11:24:09.750156 20836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:09.750468 20836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:09.750639 20836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:09.758035 20836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:09.758142 20836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:09.758258 20836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:09.767475 20836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:09.768733 20836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:09.770438 20836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:09.783785 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:09.783850 20848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:09.783874 20852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:09.784675 20836 server_base.cc:1034] running on GCE node
I20260812 11:24:09.785283 20836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:09.786247 20836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:09.787474 20836 hybrid_clock.cc:648] HybridClock initialized: now 1786533849787440 us; error 49 us; skew 500 ppm
I20260812 11:24:09.787810 20836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:09.790357 20836 webserver.cc:460] Webserver started at http://127.12.110.194:46501/ using document root <none> and password file <none>
I20260812 11:24:09.791153 20836 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:09.791296 20836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:09.791638 20836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:09.794521 20836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "aa194b31f4d541b4b2baa19143b63112"
format_stamp: "Formatted at 2026-08-12 11:24:09 on dist-test-slave-h6n0"
server_key: "139507a6f0c937670ba7b0385c211384"
server_key_iv: "39b6732e3ed8265a9981a286dc05422e"
server_key_version: "encryptionkey@0"
I20260812 11:24:09.795423 20836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "aa194b31f4d541b4b2baa19143b63112"
format_stamp: "Formatted at 2026-08-12 11:24:09 on dist-test-slave-h6n0"
server_key: "139507a6f0c937670ba7b0385c211384"
server_key_iv: "39b6732e3ed8265a9981a286dc05422e"
server_key_version: "encryptionkey@0"
I20260812 11:24:09.800943 20836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:24:09.804486 20864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:09.805882 20836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:24:09.806059 20836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "aa194b31f4d541b4b2baa19143b63112"
format_stamp: "Formatted at 2026-08-12 11:24:09 on dist-test-slave-h6n0"
server_key: "139507a6f0c937670ba7b0385c211384"
server_key_iv: "39b6732e3ed8265a9981a286dc05422e"
server_key_version: "encryptionkey@0"
I20260812 11:24:09.806286 20836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:09.811955 20836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:09.812742 20836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:09.813009 20836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:09.813970 20836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:09.815590 20836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:09.815804 20836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:09.815973 20836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:09.816073 20836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:09.851778 20836 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:43203
I20260812 11:24:09.851838 21043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:43203 every 8 connection(s)
I20260812 11:24:09.853160 20836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:09.857229 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 20836
I20260812 11:24:09.857409 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:24:09.858012 12731 external_mini_cluster.cc:1428] Setting key 39bf2d8cdae31d4d218d9a12760b39ae
I20260812 11:24:09.861202 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42099
--builtin_ntp_servers=127.12.110.212:38853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260812 11:24:09.862838 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:09.855795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:49893 (local address 127.12.110.254:42099)
0812 11:24:09.856163 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:09.856170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:09.857786 (+ 1616us) server_negotiation.cc:407] Connection header received
0812 11:24:09.859927 (+ 2141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:09.859936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:09.860007 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:09.860092 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:09.861931 (+ 1839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:09.861938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:09.861945 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:09.861968 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:09.862011 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:09.862037 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:09.862164 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:09.862375 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 11:24:09.862448 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260812 11:24:09.863696 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:09.856448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42099 (local address 127.12.110.194:49893)
0812 11:24:09.857609 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:09.857656 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:09.859613 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:09.860353 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:09.860361 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:09.860448 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:09.861391 (+ 943us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:09.861399 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:09.861498 (+ 99us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:09.861665 (+ 167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:09.862156 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:09.862166 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:09.862172 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:24:09.862409 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 11:24:09.862704 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":340,"threads_started":1}
I20260812 11:24:09.865130 21044 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42099
I20260812 11:24:09.865442 21044 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:09.866123 21044 heartbeater.cc:511] Master 127.12.110.254:42099 requested a full tablet report, sending...
I20260812 11:24:09.867717 20500 ts_manager.cc:194] Registered new tserver with Master: aa194b31f4d541b4b2baa19143b63112 (127.12.110.194:43203)
W20260812 11:24:10.199728 21049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:10.200028 21049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:10.200129 21049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:10.207306 21049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:10.207409 21049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:10.207587 21049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:10.216748 21049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:10.218158 21049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:10.220173 21049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:10.234416 21057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:10.234519 21063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:10.234573 21058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:10.236545 21049 server_base.cc:1034] running on GCE node
I20260812 11:24:10.237439 21049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:10.239048 21049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:10.240286 21049 hybrid_clock.cc:648] HybridClock initialized: now 1786533850240244 us; error 36 us; skew 500 ppm
I20260812 11:24:10.240664 21049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:10.243264 21049 webserver.cc:460] Webserver started at http://127.12.110.195:42391/ using document root <none> and password file <none>
I20260812 11:24:10.243993 21049 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:10.244089 21049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:10.244485 21049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:10.248217 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "e855bc4263634223867f7d4cf9900bfb"
format_stamp: "Formatted at 2026-08-12 11:24:10 on dist-test-slave-h6n0"
server_key: "1d97ba6dc3f7c830e45299b92b54329f"
server_key_iv: "8679d661ec8cb0250688931795c73bb3"
server_key_version: "encryptionkey@0"
I20260812 11:24:10.249220 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "e855bc4263634223867f7d4cf9900bfb"
format_stamp: "Formatted at 2026-08-12 11:24:10 on dist-test-slave-h6n0"
server_key: "1d97ba6dc3f7c830e45299b92b54329f"
server_key_iv: "8679d661ec8cb0250688931795c73bb3"
server_key_version: "encryptionkey@0"
I20260812 11:24:10.255061 21049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:24:10.259166 21071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:10.261050 21049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:24:10.261248 21049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "e855bc4263634223867f7d4cf9900bfb"
format_stamp: "Formatted at 2026-08-12 11:24:10 on dist-test-slave-h6n0"
server_key: "1d97ba6dc3f7c830e45299b92b54329f"
server_key_iv: "8679d661ec8cb0250688931795c73bb3"
server_key_version: "encryptionkey@0"
I20260812 11:24:10.261508 21049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:10.269132 21049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:10.270041 21049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:10.270380 21049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:10.271720 21049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:10.273968 21049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:10.274114 21049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:10.274246 21049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:10.274361 21049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:10.317752 21049 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:44211
I20260812 11:24:10.317857 21266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:44211 every 8 connection(s)
I20260812 11:24:10.319326 21049 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:10.320883 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 21049
I20260812 11:24:10.321122 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:24:10.321640 12731 external_mini_cluster.cc:1428] Setting key 37bd9047e9dde21ace78b393017e18b5
I20260812 11:24:10.330899 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.322569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:58745 (local address 127.12.110.254:42099)
0812 11:24:10.322774 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.322780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.324831 (+ 2051us) server_negotiation.cc:407] Connection header received
0812 11:24:10.327626 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.327630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.327696 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.327785 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.329909 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.329914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.329917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.329934 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.329981 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.329993 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.330098 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.330426 (+ 328us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.330508 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:24:10.331799 21270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.323138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42099 (local address 127.12.110.195:58745)
0812 11:24:10.324611 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.324671 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:10.327275 (+ 2604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:10.328189 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:10.328208 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:10.328353 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:10.329522 (+ 1169us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:10.329526 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:10.329590 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:10.329704 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:10.330125 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:10.330134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:10.330141 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:24:10.330390 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 11:24:10.330703 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":353,"threads_started":1}
I20260812 11:24:10.333420 21267 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42099
I20260812 11:24:10.333889 21267 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:10.334818 21267 heartbeater.cc:511] Master 127.12.110.254:42099 requested a full tablet report, sending...
I20260812 11:24:10.337247 20500 ts_manager.cc:194] Registered new tserver with Master: e855bc4263634223867f7d4cf9900bfb (127.12.110.195:44211)
I20260812 11:24:10.339205 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:24:10.349261 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.345270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.12.110.254:42099)
0812 11:24:10.345522 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.345528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.346525 (+ 997us) server_negotiation.cc:407] Connection header received
0812 11:24:10.346743 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.346749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.346880 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.347011 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.348437 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.348442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.348445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.348463 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.348498 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.348511 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.348639 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.348870 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.348988 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 11:24:10.354163 20500 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:10.361910 20500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:10.395133 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.376936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45248 (local address 127.12.110.194:43203)
0812 11:24:10.388473 (+ 11537us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.388486 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.390056 (+ 1570us) server_negotiation.cc:407] Connection header received
0812 11:24:10.390153 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.390165 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.390464 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.390666 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.393841 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.393849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.393863 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.393891 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.393953 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.393973 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.394133 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.394648 (+ 515us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.394753 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11312,"thread_start_us":12106,"threads_started":1}
I20260812 11:24:10.395732 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.375944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:44211 (local address 127.0.0.1:45552)
0812 11:24:10.384002 (+ 8058us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.384052 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:10.384242 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:10.392865 (+ 8623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:10.392876 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:10.392900 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:10.393540 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:10.393543 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:10.393574 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:10.393656 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:10.394998 (+ 1342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:10.395003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:10.395008 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:24:10.395132 (+ 124us) client_negotiation.cc:240] Negotiation successful
0812 11:24:10.395183 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7727,"thread_start_us":817,"threads_started":1}
I20260812 11:24:10.396548 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.376459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45552 (local address 127.12.110.195:44211)
0812 11:24:10.392339 (+ 15880us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.392346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.392378 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:24:10.392446 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.392454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.392657 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.393279 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.394774 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.394781 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.394792 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.394807 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.394852 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.394869 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.395989 (+ 1120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.396074 (+ 85us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.396169 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15704,"thread_start_us":15535,"threads_started":1}
I20260812 11:24:10.396569 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.376318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:43203 (local address 127.0.0.1:45248)
0812 11:24:10.389756 (+ 13438us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.389781 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:10.389913 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:10.392350 (+ 2437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:10.392361 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:10.392461 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:10.393540 (+ 1079us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:10.393543 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:10.393574 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:10.393656 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:10.394194 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:10.394208 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:10.394218 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:24:10.394410 (+ 192us) client_negotiation.cc:240] Negotiation successful
0812 11:24:10.394498 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13255,"thread_start_us":13081,"threads_started":1}
I20260812 11:24:10.398731 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.376733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46384)
0812 11:24:10.395739 (+ 19006us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.395758 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:10.395944 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:10.396675 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:10.396679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:10.396701 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:10.397426 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:10.397432 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:10.397450 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:10.397493 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:10.398052 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:10.398057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:10.398062 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:24:10.398220 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 11:24:10.398268 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18864,"thread_start_us":18696,"threads_started":1}
I20260812 11:24:10.398761 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.377433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.12.110.193:43747)
0812 11:24:10.389023 (+ 11590us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.389031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.395858 (+ 6827us) server_negotiation.cc:407] Connection header received
0812 11:24:10.396098 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.396106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.396301 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.396434 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.397662 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.397677 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.397685 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.397700 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.397787 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.397808 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.397989 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.398246 (+ 257us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.398363 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11379,"thread_start_us":11219,"threads_started":1}
I20260812 11:24:10.399861 20830 heartbeater.cc:503] Master 127.12.110.254:42099 was elected leader, sending a full tablet report...
I20260812 11:24:10.400740 20940 tablet_service.cc:1467] Processing CreateTablet for tablet af1c7aa635db4594a7467ec3bd4dd262 (DEFAULT_TABLE table=test-table [id=d8731e3166c141f78378c569db99e7fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:10.400738 21157 tablet_service.cc:1467] Processing CreateTablet for tablet af1c7aa635db4594a7467ec3bd4dd262 (DEFAULT_TABLE table=test-table [id=d8731e3166c141f78378c569db99e7fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:10.402045 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1c7aa635db4594a7467ec3bd4dd262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:10.402043 20940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1c7aa635db4594a7467ec3bd4dd262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:10.406810 20713 tablet_service.cc:1467] Processing CreateTablet for tablet af1c7aa635db4594a7467ec3bd4dd262 (DEFAULT_TABLE table=test-table [id=d8731e3166c141f78378c569db99e7fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:10.408264 20713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af1c7aa635db4594a7467ec3bd4dd262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:10.413641 21294 tablet_bootstrap.cc:492] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Bootstrap starting.
I20260812 11:24:10.416075 21295 tablet_bootstrap.cc:492] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Bootstrap starting.
I20260812 11:24:10.418387 21294 tablet_bootstrap.cc:654] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:10.419552 21295 tablet_bootstrap.cc:654] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:10.419761 21294 log.cc:824] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:10.420737 21295 log.cc:824] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:10.422046 21305 tablet_bootstrap.cc:492] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: Bootstrap starting.
I20260812 11:24:10.423484 21294 tablet_bootstrap.cc:492] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: No bootstrap required, opened a new log
I20260812 11:24:10.423763 21294 ts_tablet_manager.cc:1397] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260812 11:24:10.426149 21295 tablet_bootstrap.cc:492] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: No bootstrap required, opened a new log
I20260812 11:24:10.426179 21305 tablet_bootstrap.cc:654] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:10.426445 21295 ts_tablet_manager.cc:1397] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260812 11:24:10.427461 21305 log.cc:824] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:10.431946 21294 raft_consensus.cc:348] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.432258 21305 tablet_bootstrap.cc:492] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: No bootstrap required, opened a new log
I20260812 11:24:10.432511 21305 ts_tablet_manager.cc:1397] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260812 11:24:10.432513 21294 raft_consensus.cc:374] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:10.432622 21294 raft_consensus.cc:729] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e855bc4263634223867f7d4cf9900bfb, State: Initialized, Role: FOLLOWER
I20260812 11:24:10.433475 21294 consensus_queue.cc:260] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.434639 21267 heartbeater.cc:503] Master 127.12.110.254:42099 was elected leader, sending a full tablet report...
I20260812 11:24:10.435408 21294 ts_tablet_manager.cc:1428] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
I20260812 11:24:10.438597 21295 raft_consensus.cc:348] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.439446 21295 raft_consensus.cc:374] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:10.439651 21295 raft_consensus.cc:729] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa194b31f4d541b4b2baa19143b63112, State: Initialized, Role: FOLLOWER
I20260812 11:24:10.440662 21295 consensus_queue.cc:260] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.442130 21044 heartbeater.cc:503] Master 127.12.110.254:42099 was elected leader, sending a full tablet report...
I20260812 11:24:10.442929 21295 ts_tablet_manager.cc:1428] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Time spent starting tablet: real 0.016s user 0.010s sys 0.006s
I20260812 11:24:10.444535 21305 raft_consensus.cc:348] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.445340 21305 raft_consensus.cc:374] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:10.445466 21305 raft_consensus.cc:729] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4150fd358f4d45418c016ca3d0044b10, State: Initialized, Role: FOLLOWER
I20260812 11:24:10.446102 21305 consensus_queue.cc:260] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.448990 21305 ts_tablet_manager.cc:1428] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260812 11:24:10.571694 21269 tablet.cc:2369] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:24:10.605437 21045 tablet.cc:2369] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:24:10.633248 20831 tablet.cc:2369] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:10.657544 21308 raft_consensus.cc:482] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:10.658346 21308 raft_consensus.cc:504] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.660240 21308 leader_election.cc:288] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e855bc4263634223867f7d4cf9900bfb (127.12.110.195:44211), 4150fd358f4d45418c016ca3d0044b10 (127.12.110.193:43747)
I20260812 11:24:10.663760 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.660782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:32917 (local address 127.12.110.195:44211)
0812 11:24:10.661026 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.661035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.661801 (+ 766us) server_negotiation.cc:407] Connection header received
0812 11:24:10.661992 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.661997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.662077 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.662178 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.663137 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.663142 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.663147 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.663178 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.663207 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.663219 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.663369 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.663425 (+ 56us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.663492 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:24:10.663965 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.660708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:33693 (local address 127.12.110.193:43747)
0812 11:24:10.660982 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.660992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.661720 (+ 728us) server_negotiation.cc:407] Connection header received
0812 11:24:10.661893 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.661901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.661968 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.662054 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.663327 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.663332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.663336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.663354 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.663391 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.663408 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.663528 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.663643 (+ 115us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.663699 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:24:10.664009 21315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.660645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:44211 (local address 127.12.110.194:32917)
0812 11:24:10.661703 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.661722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:10.661843 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:10.662232 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:10.662235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:10.662253 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:10.662935 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:10.662937 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:10.662951 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:10.662989 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:10.663328 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:10.663331 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:10.663334 (+ 3us) client_negotiation.cc:769] Sending connection context
0812 11:24:10.663431 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 11:24:10.663508 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":669,"threads_started":1}
I20260812 11:24:10.664201 21312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.660609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.12.110.194:33693)
0812 11:24:10.661585 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.661614 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:10.661734 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:10.662122 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:10.662129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:10.662158 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:10.663137 (+ 979us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:24:10.663140 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:10.663152 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:24:10.663186 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:10.663528 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:10.663534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:10.663545 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 11:24:10.663687 (+ 142us) client_negotiation.cc:240] Negotiation successful
0812 11:24:10.663732 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":643,"threads_started":1}
I20260812 11:24:10.665251 21187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1c7aa635db4594a7467ec3bd4dd262" candidate_uuid: "aa194b31f4d541b4b2baa19143b63112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e855bc4263634223867f7d4cf9900bfb" is_pre_election: true
I20260812 11:24:10.665244 20759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1c7aa635db4594a7467ec3bd4dd262" candidate_uuid: "aa194b31f4d541b4b2baa19143b63112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4150fd358f4d45418c016ca3d0044b10" is_pre_election: true
I20260812 11:24:10.665732 20759 raft_consensus.cc:2454] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa194b31f4d541b4b2baa19143b63112 in term 0.
I20260812 11:24:10.665731 21187 raft_consensus.cc:2454] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa194b31f4d541b4b2baa19143b63112 in term 0.
I20260812 11:24:10.666421 20867 leader_election.cc:302] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4150fd358f4d45418c016ca3d0044b10, aa194b31f4d541b4b2baa19143b63112; no voters:
I20260812 11:24:10.666844 21308 raft_consensus.cc:2781] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:10.667029 21308 raft_consensus.cc:482] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:10.667150 21308 raft_consensus.cc:3037] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:10.669652 21308 raft_consensus.cc:504] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.670351 21308 leader_election.cc:288] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [CANDIDATE]: Term 1 election: Requested vote from peers e855bc4263634223867f7d4cf9900bfb (127.12.110.195:44211), 4150fd358f4d45418c016ca3d0044b10 (127.12.110.193:43747)
I20260812 11:24:10.670956 20759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1c7aa635db4594a7467ec3bd4dd262" candidate_uuid: "aa194b31f4d541b4b2baa19143b63112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4150fd358f4d45418c016ca3d0044b10"
I20260812 11:24:10.671188 20759 raft_consensus.cc:3037] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:10.671389 21187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af1c7aa635db4594a7467ec3bd4dd262" candidate_uuid: "aa194b31f4d541b4b2baa19143b63112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e855bc4263634223867f7d4cf9900bfb"
I20260812 11:24:10.671779 21187 raft_consensus.cc:3037] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:10.673595 20759 raft_consensus.cc:2454] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa194b31f4d541b4b2baa19143b63112 in term 1.
I20260812 11:24:10.674082 20867 leader_election.cc:302] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4150fd358f4d45418c016ca3d0044b10, aa194b31f4d541b4b2baa19143b63112; no voters:
I20260812 11:24:10.674190 21187 raft_consensus.cc:2454] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa194b31f4d541b4b2baa19143b63112 in term 1.
I20260812 11:24:10.674399 21308 raft_consensus.cc:2781] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:10.674934 21308 raft_consensus.cc:686] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 1 LEADER]: Becoming Leader. State: Replica: aa194b31f4d541b4b2baa19143b63112, State: Running, Role: LEADER
I20260812 11:24:10.675485 21308 consensus_queue.cc:237] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } }
I20260812 11:24:10.680395 20499 catalog_manager.cc:5495] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa194b31f4d541b4b2baa19143b63112 (127.12.110.194). New cstate: current_term: 1 leader_uuid: "aa194b31f4d541b4b2baa19143b63112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa194b31f4d541b4b2baa19143b63112" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 } health_report { overall_health: UNKNOWN } } }
I20260812 11:24:10.701514 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.698624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.12.110.194:43203)
0812 11:24:10.698834 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.698842 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.698869 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:24:10.699246 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.699252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.699382 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.699527 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.700560 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.700566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.700569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.700587 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.700629 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.700641 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.700873 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.701117 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.701173 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:24:10.712201 20759 raft_consensus.cc:1261] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10 [term 1 FOLLOWER]: Refusing update from remote peer aa194b31f4d541b4b2baa19143b63112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:10.712272 21187 raft_consensus.cc:1261] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 1 FOLLOWER]: Refusing update from remote peer aa194b31f4d541b4b2baa19143b63112: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:10.713212 21308 consensus_queue.cc:1035] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4150fd358f4d45418c016ca3d0044b10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:10.713523 21317 consensus_queue.cc:1035] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e855bc4263634223867f7d4cf9900bfb" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:10.729488 21322 mvcc.cc:205] Tried to move back new op lower bound from 7317642652505591808 to 7317642652370923520. Current Snapshot: MvccSnapshot[applied={T|T < 7317642652505591808}]
I20260812 11:24:10.732079 21324 mvcc.cc:205] Tried to move back new op lower bound from 7317642652505591808 to 7317642652370923520. Current Snapshot: MvccSnapshot[applied={T|T < 7317642652505591808}]
I20260812 11:24:10.743667 20499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:10.744037 20499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:10.748404 20499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P faa04bced8d14e7893ca131f43f8af14: Sending DeleteTablet for 3 replicas of tablet af1c7aa635db4594a7467ec3bd4dd262
I20260812 11:24:10.749770 20940 tablet_service.cc:1514] Processing DeleteTablet for tablet af1c7aa635db4594a7467ec3bd4dd262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:10 UTC) from {username='slave'} at 127.0.0.1:45248
I20260812 11:24:10.749893 21157 tablet_service.cc:1514] Processing DeleteTablet for tablet af1c7aa635db4594a7467ec3bd4dd262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:10 UTC) from {username='slave'} at 127.0.0.1:45552
I20260812 11:24:10.749922 20713 tablet_service.cc:1514] Processing DeleteTablet for tablet af1c7aa635db4594a7467ec3bd4dd262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:10 UTC) from {username='slave'} at 127.0.0.1:46384
I20260812 11:24:10.751675 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 20605
I20260812 11:24:10.759559 21341 tablet_replica.cc:330] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: stopping tablet replica
I20260812 11:24:10.760109 21341 raft_consensus.cc:2229] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:10.760701 21341 raft_consensus.cc:2258] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:10.761853 21293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.752914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.12.110.193:43747)
0812 11:24:10.753158 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.753168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.759004 (+ 5836us) server_negotiation.cc:407] Connection header received
0812 11:24:10.759198 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.759208 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.759376 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.759518 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.760734 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.760741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.760746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.760770 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.760815 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.760832 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.760980 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.761437 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.761507 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:24:10.763780 20825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:10.766026 21341 ts_tablet_manager.cc:1905] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:10.771338 21339 tablet_replica.cc:330] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: stopping tablet replica
I20260812 11:24:10.771929 21339 raft_consensus.cc:2229] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:10.772789 21339 raft_consensus.cc:2258] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:10.775457 21339 ts_tablet_manager.cc:1905] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:10.779942 21341 ts_tablet_manager.cc:1918] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:10.780143 21341 log.cc:1196] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/af1c7aa635db4594a7467ec3bd4dd262
I20260812 11:24:10.780786 21341 ts_tablet_manager.cc:1939] T af1c7aa635db4594a7467ec3bd4dd262 P e855bc4263634223867f7d4cf9900bfb: Deleting consensus metadata
I20260812 11:24:10.782996 21339 ts_tablet_manager.cc:1918] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:10.782954 20485 catalog_manager.cc:4841] TS e855bc4263634223867f7d4cf9900bfb (127.12.110.195:44211): tablet af1c7aa635db4594a7467ec3bd4dd262 (table test-table [id=d8731e3166c141f78378c569db99e7fd]) successfully deleted
I20260812 11:24:10.783174 21339 log.cc:1196] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/af1c7aa635db4594a7467ec3bd4dd262
I20260812 11:24:10.783882 21339 ts_tablet_manager.cc:1939] T af1c7aa635db4594a7467ec3bd4dd262 P aa194b31f4d541b4b2baa19143b63112: Deleting consensus metadata
I20260812 11:24:10.786018 20484 catalog_manager.cc:4841] TS aa194b31f4d541b4b2baa19143b63112 (127.12.110.194:43203): tablet af1c7aa635db4594a7467ec3bd4dd262 (table test-table [id=d8731e3166c141f78378c569db99e7fd]) successfully deleted
I20260812 11:24:10.908284 21340 tablet_replica.cc:330] T af1c7aa635db4594a7467ec3bd4dd262 P 4150fd358f4d45418c016ca3d0044b10: stopping tablet replica
I20260812 11:24:10.908533 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 20605
I20260812 11:24:10.937841 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 20836
W20260812 11:24:10.937783 20482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.110.193:43747 (error 108)
I20260812 11:24:10.940402 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.939320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46398)
0812 11:24:10.940073 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.940195 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":499,"threads_started":1}
W20260812 11:24:10.940610 20482 catalog_manager.cc:4584] TS 4150fd358f4d45418c016ca3d0044b10 (127.12.110.193:43747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af1c7aa635db4594a7467ec3bd4dd262: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
I20260812 11:24:10.941195 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.938685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45258 (local address 127.12.110.194:43203)
0812 11:24:10.938875 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:10.938882 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:10.938985 (+ 103us) server_negotiation.cc:407] Connection header received
0812 11:24:10.939203 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:10.939211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:10.939285 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:10.939408 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:10.940416 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:10.940421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:10.940426 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:10.940481 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:10.940529 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:10.940549 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:10.940668 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:10.940830 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 11:24:10.940891 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:24:10.941941 21033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:10.994705 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.994102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46410)
0812 11:24:10.994352 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.994485 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 11:24:10.995887 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:10.995490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46424)
0812 11:24:10.995645 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:10.995725 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260812 11:24:11.057209 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:11.056586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46430)
0812 11:24:11.056866 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:11.057007 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260812 11:24:11.058488 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:11.058065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46444)
0812 11:24:11.058230 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:11.058320 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 11:24:11.063520 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 20836
I20260812 11:24:11.092095 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 21049
I20260812 11:24:11.095501 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:11.093058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.12.110.195:44211)
0812 11:24:11.093263 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:11.093268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:11.093398 (+ 130us) server_negotiation.cc:407] Connection header received
0812 11:24:11.093614 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:11.093618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:11.093709 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:11.093807 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:11.094849 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:11.094853 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:11.094856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:11.094872 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:11.094906 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:11.094918 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:11.095055 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:11.095218 (+ 163us) server_negotiation.cc:299] Negotiation successful
0812 11:24:11.095269 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:24:11.096179 21260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:11.139755 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:11.139120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46454)
0812 11:24:11.139441 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:11.139568 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260812 11:24:11.141489 21348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:11.140957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43747 (local address 127.0.0.1:46464)
0812 11:24:11.141224 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:11.141326 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:43747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260812 11:24:11.188443 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 21049
I20260812 11:24:11.209901 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 20458
I20260812 11:24:11.211478 20557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:11.322214 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 20458
2026-08-12T11:24:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3080 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260812 11:24:11.380352 12731 mini_oidc.cc:152] Starting JWKS server
I20260812 11:24:11.383250 12731 webserver.cc:460] Webserver started at https://127.0.0.1:34643/ using document root <none> and password file <none>
I20260812 11:24:11.383597 12731 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:24:11.384279 12731 webserver.cc:460] Webserver started at http://127.0.0.1:43933/ using document root <none> and password file <none>
2026-08-12T11:24:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:11Z Disabled control of system clock
I20260812 11:24:11.425446 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42885
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:43701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:42885
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34643/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/testchainca.cert with env {}
W20260812 11:24:11.774206 21363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:11.774569 21363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:11.774730 21363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:11.781908 21363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:11.782054 21363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:11.782147 21363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:11.782255 21363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:11.782341 21363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:24:11.782434 21363 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:24:11.782513 21363 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34643/jwks.json
I20260812 11:24:11.794359 21363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:43701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:42885
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34643/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:11.795761 21363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:11.797096 21363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:11.809368 21369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:11.809283 21372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:11.809329 21370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:11.810146 21363 server_base.cc:1034] running on GCE node
I20260812 11:24:11.810936 21363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:11.812225 21363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:11.813462 21363 hybrid_clock.cc:648] HybridClock initialized: now 1786533851813416 us; error 45 us; skew 500 ppm
I20260812 11:24:11.813824 21363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:11.824133 21363 webserver.cc:460] Webserver started at http://127.12.110.254:37667/ using document root <none> and password file <none>
I20260812 11:24:11.824904 21363 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:11.825057 21363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:11.825388 21363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:11.828457 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "e912b31793cc49c3ab2dbb69ba3ee525"
format_stamp: "Formatted at 2026-08-12 11:24:11 on dist-test-slave-h6n0"
server_key: "349405963e5828eee39197368b22f7e5"
server_key_iv: "b033f18cb05410230d67a61d0ce916ee"
server_key_version: "encryptionkey@0"
I20260812 11:24:11.829259 21363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "e912b31793cc49c3ab2dbb69ba3ee525"
format_stamp: "Formatted at 2026-08-12 11:24:11 on dist-test-slave-h6n0"
server_key: "349405963e5828eee39197368b22f7e5"
server_key_iv: "b033f18cb05410230d67a61d0ce916ee"
server_key_version: "encryptionkey@0"
I20260812 11:24:11.834626 21363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260812 11:24:11.838063 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:11.839469 21363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:24:11.839681 21363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "e912b31793cc49c3ab2dbb69ba3ee525"
format_stamp: "Formatted at 2026-08-12 11:24:11 on dist-test-slave-h6n0"
server_key: "349405963e5828eee39197368b22f7e5"
server_key_iv: "b033f18cb05410230d67a61d0ce916ee"
server_key_version: "encryptionkey@0"
I20260812 11:24:11.839910 21363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:11.866773 21363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:11.867698 21363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:11.868043 21363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:11.887310 21363 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:42885
I20260812 11:24:11.887347 21469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:42885 every 8 connection(s)
I20260812 11:24:11.888677 21363 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:11.892333 21473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:11.894784 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 21363
I20260812 11:24:11.895020 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:11.895663 12731 external_mini_cluster.cc:1428] Setting key 1ebe2fbc147202c4c9bbbd1ca108ddcf
I20260812 11:24:11.904739 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525: Bootstrap starting.
I20260812 11:24:11.908514 21473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:11.909644 21473 log.cc:824] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:11.910135 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:11.897636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.12.110.254:42885)
0812 11:24:11.898785 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:11.898795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:11.898836 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:24:11.900510 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:11.900530 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:11.900965 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:11.901286 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:11.901896 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:11.903275 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:11.905430 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:11.906099 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:11.906250 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:11.907678 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:11.907685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:11.907697 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:11.908217 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:11.908304 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:11.908325 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:11.908428 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:11.908761 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 11:24:11.909109 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":508,"threads_started":1}
I20260812 11:24:11.913954 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:24:11.914592 21473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525: No bootstrap required, opened a new log
I20260812 11:24:11.922264 21473 raft_consensus.cc:348] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42885 } }
I20260812 11:24:11.922746 21473 raft_consensus.cc:374] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:11.922876 21473 raft_consensus.cc:729] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e912b31793cc49c3ab2dbb69ba3ee525, State: Initialized, Role: FOLLOWER
I20260812 11:24:11.923444 21473 consensus_queue.cc:260] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42885 } }
I20260812 11:24:11.923696 21473 raft_consensus.cc:388] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:11.923841 21473 raft_consensus.cc:482] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:11.924026 21473 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:11.926213 21473 raft_consensus.cc:504] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42885 } }
I20260812 11:24:11.926831 21473 leader_election.cc:302] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e912b31793cc49c3ab2dbb69ba3ee525; no voters:
I20260812 11:24:11.927425 21473 leader_election.cc:288] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:11.927625 21478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:11.928377 21478 raft_consensus.cc:686] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [term 1 LEADER]: Becoming Leader. State: Replica: e912b31793cc49c3ab2dbb69ba3ee525, State: Running, Role: LEADER
I20260812 11:24:11.928944 21478 consensus_queue.cc:237] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42885 } }
I20260812 11:24:11.929786 21473 sys_catalog.cc:564] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:11.932153 21482 sys_catalog.cc:455] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e912b31793cc49c3ab2dbb69ba3ee525. Latest consensus state: current_term: 1 leader_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42885 } } }
I20260812 11:24:11.932361 21479 sys_catalog.cc:455] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e912b31793cc49c3ab2dbb69ba3ee525" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42885 } } }
I20260812 11:24:11.932663 21482 sys_catalog.cc:458] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:11.932689 21479 sys_catalog.cc:458] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:11.934708 21493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:11.939800 21493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:11.948802 21493 catalog_manager.cc:1347] Generated new cluster ID: 3e34eb2a335f4dff889c04d7ed8cb4cc
I20260812 11:24:11.948999 21493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:11.956605 21493 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:11.958359 21493 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:11.967466 21493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e912b31793cc49c3ab2dbb69ba3ee525: Generated new TSK 0
I20260812 11:24:11.968616 21493 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 11:24:12.008591 21520 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260812 11:24:12.008829 21520 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 11:24:12.010138 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:11.986475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.12.110.254:42885)
0812 11:24:11.986675 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:11.986683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:11.987048 (+ 365us) server_negotiation.cc:407] Connection header received
0812 11:24:11.987360 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:11.987369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:11.987473 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:11.987642 (+ 169us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:24:11.988188 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:11.989314 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:11.990796 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:11.991485 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:11.991623 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:11.992684 (+ 1061us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:24:12.009368 (+ 16684us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 11:24:12.009782 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":77}
W20260812 11:24:12.010280 21476 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45574: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260812 11:24:12.011672 21516 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:11.986251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42885 (local address 127.0.0.1:45574)
0812 11:24:11.986944 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:11.986959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:11.987105 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:11.987686 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:11.987690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:11.987699 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:24:11.988029 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:11.988037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:11.989459 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:11.989463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:11.990658 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:11.990670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:11.991843 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:11.991850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:11.992047 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:11.992288 (+ 241us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:24:12.009674 (+ 17386us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0812 11:24:12.009960 (+ 286us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.110.254:42885: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":376,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.12.110.254:42885: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34643/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260812 11:24:12.013679 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 21363
I20260812 11:24:12.015067 21455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:12.110898 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 21363
2026-08-12T11:24:12Z chronyd exiting
I20260812 11:24:12.173418 12731 test_util.cc:175] -----------------------------------------------
I20260812 11:24:12.173617 12731 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786533802845873-12731-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (798 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260812 11:24:12.178056 12731 mini_oidc.cc:152] Starting JWKS server
I20260812 11:24:12.180732 12731 webserver.cc:460] Webserver started at https://127.0.0.1:46285/ using document root <none> and password file <none>
I20260812 11:24:12.180981 12731 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:24:12.181810 12731 webserver.cc:460] Webserver started at http://127.0.0.1:40427/ using document root <none> and password file <none>
2026-08-12T11:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:12Z Disabled control of system clock
I20260812 11:24:12.220369 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41315
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:41315
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46285/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/testchainca.cert with env {}
W20260812 11:24:12.581977 21530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:12.582398 21530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:12.582608 21530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:12.590209 21530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:12.590368 21530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:12.590440 21530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:12.590548 21530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:12.590617 21530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:24:12.590705 21530 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:24:12.590790 21530 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46285/jwks.json
I20260812 11:24:12.601243 21530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:41315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46285/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:12.602761 21530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:12.604132 21530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:12.616886 21536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:12.616981 21539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:12.617002 21537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:12.617836 21530 server_base.cc:1034] running on GCE node
I20260812 11:24:12.618767 21530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:12.620200 21530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:12.621480 21530 hybrid_clock.cc:648] HybridClock initialized: now 1786533852621441 us; error 48 us; skew 500 ppm
I20260812 11:24:12.621891 21530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:12.624591 21530 webserver.cc:460] Webserver started at http://127.12.110.254:36215/ using document root <none> and password file <none>
I20260812 11:24:12.625370 21530 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:12.625527 21530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:12.625917 21530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:12.629002 21530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "b16fedd2032645ec81e59a09f22b6181"
format_stamp: "Formatted at 2026-08-12 11:24:12 on dist-test-slave-h6n0"
server_key: "0e3ad0af948ca2c200cec5b6f9973751"
server_key_iv: "14f85c89c9d11b31507e5301f3e6749a"
server_key_version: "encryptionkey@0"
I20260812 11:24:12.629870 21530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "b16fedd2032645ec81e59a09f22b6181"
format_stamp: "Formatted at 2026-08-12 11:24:12 on dist-test-slave-h6n0"
server_key: "0e3ad0af948ca2c200cec5b6f9973751"
server_key_iv: "14f85c89c9d11b31507e5301f3e6749a"
server_key_version: "encryptionkey@0"
I20260812 11:24:12.635234 21530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:24:12.638651 21549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:12.640053 21530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:24:12.640358 21530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "b16fedd2032645ec81e59a09f22b6181"
format_stamp: "Formatted at 2026-08-12 11:24:12 on dist-test-slave-h6n0"
server_key: "0e3ad0af948ca2c200cec5b6f9973751"
server_key_iv: "14f85c89c9d11b31507e5301f3e6749a"
server_key_version: "encryptionkey@0"
I20260812 11:24:12.640594 21530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:12.668434 21530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:12.669297 21530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:12.669634 21530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:12.690227 21530 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:41315
I20260812 11:24:12.690253 21631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:41315 every 8 connection(s)
I20260812 11:24:12.691707 21530 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:12.695307 21632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:12.696877 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 21530
I20260812 11:24:12.697103 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:12.697942 12731 external_mini_cluster.cc:1428] Setting key 2410fa85bea688e82ae4ef9cd3bd1d7b
I20260812 11:24:12.708949 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181: Bootstrap starting.
I20260812 11:24:12.712428 21632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:12.713539 21632 log.cc:824] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:12.714298 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:12.700278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.12.110.254:41315)
0812 11:24:12.701578 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:12.701592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:12.701720 (+ 128us) server_negotiation.cc:407] Connection header received
0812 11:24:12.704119 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:12.704146 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:12.704707 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:12.705148 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:12.705688 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:12.707745 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:12.709049 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:12.710038 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:12.710234 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:12.711862 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:12.711869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:12.711882 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:12.712393 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:12.712544 (+ 151us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:12.712579 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:12.712706 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:12.712935 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 11:24:12.713222 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":476,"threads_started":1}
I20260812 11:24:12.716377 21632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181: No bootstrap required, opened a new log
I20260812 11:24:12.718246 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:24:12.724773 21632 raft_consensus.cc:348] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16fedd2032645ec81e59a09f22b6181" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41315 } }
I20260812 11:24:12.725092 21632 raft_consensus.cc:374] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:12.725165 21632 raft_consensus.cc:729] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16fedd2032645ec81e59a09f22b6181, State: Initialized, Role: FOLLOWER
I20260812 11:24:12.725638 21632 consensus_queue.cc:260] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16fedd2032645ec81e59a09f22b6181" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41315 } }
I20260812 11:24:12.725843 21632 raft_consensus.cc:388] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:12.726002 21632 raft_consensus.cc:482] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:12.726119 21632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:12.728539 21632 raft_consensus.cc:504] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16fedd2032645ec81e59a09f22b6181" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41315 } }
I20260812 11:24:12.728984 21632 leader_election.cc:302] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b16fedd2032645ec81e59a09f22b6181; no voters:
I20260812 11:24:12.729472 21632 leader_election.cc:288] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:12.729622 21643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:12.730352 21643 raft_consensus.cc:686] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [term 1 LEADER]: Becoming Leader. State: Replica: b16fedd2032645ec81e59a09f22b6181, State: Running, Role: LEADER
I20260812 11:24:12.730902 21643 consensus_queue.cc:237] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16fedd2032645ec81e59a09f22b6181" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41315 } }
I20260812 11:24:12.731513 21632 sys_catalog.cc:564] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:12.733377 21651 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b16fedd2032645ec81e59a09f22b6181. Latest consensus state: current_term: 1 leader_uuid: "b16fedd2032645ec81e59a09f22b6181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16fedd2032645ec81e59a09f22b6181" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41315 } } }
I20260812 11:24:12.733697 21651 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:12.734809 21645 sys_catalog.cc:455] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b16fedd2032645ec81e59a09f22b6181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b16fedd2032645ec81e59a09f22b6181" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41315 } } }
I20260812 11:24:12.735208 21645 sys_catalog.cc:458] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:12.739595 21657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:12.744325 21657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:12.754961 21657 catalog_manager.cc:1347] Generated new cluster ID: 15c4c6f69856498a9c45261b374ce2b3
I20260812 11:24:12.755103 21657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:12.782696 21657 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:12.784925 21657 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:12.797758 21657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b16fedd2032645ec81e59a09f22b6181: Generated new TSK 0
I20260812 11:24:12.798950 21657 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260812 11:24:12.842020 21687 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260812 11:24:12.842177 21687 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 11:24:12.843398 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:12.821807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.12.110.254:41315)
0812 11:24:12.822068 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:12.822078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:12.822482 (+ 404us) server_negotiation.cc:407] Connection header received
0812 11:24:12.822890 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:12.822897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:12.823014 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:12.823182 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:24:12.823629 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:12.824894 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:12.826912 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:12.827618 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:12.827753 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:12.828368 (+ 615us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:24:12.842709 (+ 14341us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 11:24:12.843083 (+ 374us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260812 11:24:12.843538 21640 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260812 11:24:12.843832 21686 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:12.821503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41315 (local address 127.0.0.1:51094)
0812 11:24:12.822370 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:12.822390 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:12.822506 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:12.823210 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:12.823214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:12.823224 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:24:12.823461 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:12.823470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:12.825188 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:12.825194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:12.826745 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:12.826761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:12.827817 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:12.827824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:12.828022 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:12.828086 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:24:12.843020 (+ 14934us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0812 11:24:12.843260 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.110.254:41315: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46285/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":602,"threads_started":1}
I20260812 11:24:12.845890 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 21530
I20260812 11:24:12.847230 21624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:12.947021 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 21530
2026-08-12T11:24:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (839 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260812 11:24:13.017670 12731 mini_oidc.cc:152] Starting JWKS server
I20260812 11:24:13.020032 12731 webserver.cc:460] Webserver started at https://127.0.0.1:37791/ using document root <none> and password file <none>
I20260812 11:24:13.020326 12731 mini_oidc.cc:189] Starting OIDC Discovery server
I20260812 11:24:13.021036 12731 webserver.cc:460] Webserver started at http://127.0.0.1:41963/ using document root <none> and password file <none>
2026-08-12T11:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:13Z Disabled control of system clock
I20260812 11:24:13.059288 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:39985
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:44869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:39985
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37791/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 11:24:13.410686 21696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:13.411029 21696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:13.411190 21696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:13.418262 21696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:13.418401 21696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:13.418493 21696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:13.418620 21696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:13.418708 21696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:24:13.418798 21696 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:24:13.418876 21696 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37791/jwks.json
I20260812 11:24:13.428117 21696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:44869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:39985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37791/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:13.429400 21696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:13.431236 21696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:13.442857 21712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:13.443172 21707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:13.443895 21706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:13.445343 21696 server_base.cc:1034] running on GCE node
I20260812 11:24:13.446168 21696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:13.447597 21696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:13.448849 21696 hybrid_clock.cc:648] HybridClock initialized: now 1786533853448802 us; error 37 us; skew 500 ppm
I20260812 11:24:13.449229 21696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:13.451751 21696 webserver.cc:460] Webserver started at http://127.12.110.254:45715/ using document root <none> and password file <none>
I20260812 11:24:13.452481 21696 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:13.452617 21696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:13.452955 21696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:13.456043 21696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "9cbd08651373458aa5babb27c74c28f3"
format_stamp: "Formatted at 2026-08-12 11:24:13 on dist-test-slave-h6n0"
server_key: "d0c535f778ec61cca81aaa0215bf1cae"
server_key_iv: "fc328f8e2db3d2d12c85dedc469a6af7"
server_key_version: "encryptionkey@0"
I20260812 11:24:13.456841 21696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "9cbd08651373458aa5babb27c74c28f3"
format_stamp: "Formatted at 2026-08-12 11:24:13 on dist-test-slave-h6n0"
server_key: "d0c535f778ec61cca81aaa0215bf1cae"
server_key_iv: "fc328f8e2db3d2d12c85dedc469a6af7"
server_key_version: "encryptionkey@0"
I20260812 11:24:13.461804 21696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:24:13.465194 21720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:13.466405 21696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:24:13.466648 21696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "9cbd08651373458aa5babb27c74c28f3"
format_stamp: "Formatted at 2026-08-12 11:24:13 on dist-test-slave-h6n0"
server_key: "d0c535f778ec61cca81aaa0215bf1cae"
server_key_iv: "fc328f8e2db3d2d12c85dedc469a6af7"
server_key_version: "encryptionkey@0"
I20260812 11:24:13.466902 21696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:13.489105 21696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:13.489995 21696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:13.490324 21696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:13.511018 21696 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:39985
I20260812 11:24:13.511056 21809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:39985 every 8 connection(s)
I20260812 11:24:13.512410 21696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:13.514020 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 21696
I20260812 11:24:13.514246 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:13.515116 12731 external_mini_cluster.cc:1428] Setting key faef1fdd52c64be6823080283f953684
I20260812 11:24:13.517489 21811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:13.531118 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:13.517148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.12.110.254:39985)
0812 11:24:13.518573 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:13.518588 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:13.518638 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:24:13.520762 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:13.520795 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:13.521384 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:13.521841 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:13.522385 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:13.524459 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:13.525860 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:13.526908 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:13.527083 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:13.528325 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:13.528340 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:13.528357 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:13.529106 (+ 749us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:13.529202 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:13.529227 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:13.529337 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:13.529622 (+ 285us) server_negotiation.cc:299] Negotiation successful
0812 11:24:13.529963 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":553,"threads_started":1}
I20260812 11:24:13.532542 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3: Bootstrap starting.
I20260812 11:24:13.535061 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:24:13.536017 21811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:13.537050 21811 log.cc:824] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:13.540056 21811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3: No bootstrap required, opened a new log
I20260812 11:24:13.547775 21811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd08651373458aa5babb27c74c28f3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39985 } }
I20260812 11:24:13.548178 21811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:13.548295 21811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cbd08651373458aa5babb27c74c28f3, State: Initialized, Role: FOLLOWER
I20260812 11:24:13.548852 21811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd08651373458aa5babb27c74c28f3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39985 } }
I20260812 11:24:13.549157 21811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:13.549295 21811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:13.549469 21811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:13.551764 21811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd08651373458aa5babb27c74c28f3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39985 } }
I20260812 11:24:13.552332 21811 leader_election.cc:302] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cbd08651373458aa5babb27c74c28f3; no voters:
I20260812 11:24:13.552897 21811 leader_election.cc:288] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:13.553063 21821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:13.554126 21821 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [term 1 LEADER]: Becoming Leader. State: Replica: 9cbd08651373458aa5babb27c74c28f3, State: Running, Role: LEADER
I20260812 11:24:13.554828 21821 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd08651373458aa5babb27c74c28f3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39985 } }
I20260812 11:24:13.555230 21811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:13.557960 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cbd08651373458aa5babb27c74c28f3. Latest consensus state: current_term: 1 leader_uuid: "9cbd08651373458aa5babb27c74c28f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd08651373458aa5babb27c74c28f3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39985 } } }
I20260812 11:24:13.558357 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:13.558794 21823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cbd08651373458aa5babb27c74c28f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cbd08651373458aa5babb27c74c28f3" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39985 } } }
I20260812 11:24:13.559178 21823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:13.571085 21830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:13.576014 21830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:13.585352 21830 catalog_manager.cc:1347] Generated new cluster ID: f53dd7faec794d27a6f12452db3e32fd
I20260812 11:24:13.585487 21830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:13.605427 21830 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:13.607322 21830 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:13.622503 21830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cbd08651373458aa5babb27c74c28f3: Generated new TSK 0
I20260812 11:24:13.623440 21830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:24:13.649816 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:13.640168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.12.110.254:39985)
0812 11:24:13.640435 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:13.640441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:13.640893 (+ 452us) server_negotiation.cc:407] Connection header received
0812 11:24:13.641102 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:13.641107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:13.641338 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:13.641490 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:24:13.642133 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:13.642932 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:13.645228 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:13.645710 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:13.645820 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:13.646795 (+ 975us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:24:13.649185 (+ 2390us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 11:24:13.649502 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 11:24:13.649945 21818 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46142: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260812 11:24:13.651934 21868 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:13.640048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:39985 (local address 127.0.0.1:46142)
0812 11:24:13.640701 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:13.640720 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:13.640879 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:13.641544 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:13.641549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:13.641562 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:24:13.641936 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:13.641949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:13.643094 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:13.643099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:13.644848 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:13.644865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:13.646031 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:13.646041 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:13.646424 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:13.646478 (+ 54us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:24:13.649406 (+ 2928us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0812 11:24:13.651479 (+ 2073us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.110.254:39985: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37791/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":1732,"thread_start_us":333,"threads_started":1}
I20260812 11:24:13.652972 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 21696
I20260812 11:24:13.654727 21804 generic_service.cc:196] Checking for leaks (request via RPC)
E20260812 11:24:13.661965 21869 webserver.cc:533] Webserver: sslize failed:
E20260812 11:24:13.662057 21869 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260812 11:24:13.747213 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 21696
2026-08-12T11:24:13Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (795 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-12T11:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:13Z Disabled control of system clock
I20260812 11:24:13.849043 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43853
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:35037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:43853
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/testchainca.cert with env {}
W20260812 11:24:14.198290 21885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:14.198638 21885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:14.198747 21885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:14.206353 21885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:14.206451 21885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:14.206502 21885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:14.206622 21885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:14.206672 21885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260812 11:24:14.206728 21885 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260812 11:24:14.206768 21885 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260812 11:24:14.216419 21885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:43853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:14.217672 21885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:14.218986 21885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:14.231000 21894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:14.230983 21896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:14.231065 21899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:14.232586 21885 server_base.cc:1034] running on GCE node
I20260812 11:24:14.233413 21885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:14.234733 21885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:14.235958 21885 hybrid_clock.cc:648] HybridClock initialized: now 1786533854235906 us; error 41 us; skew 500 ppm
I20260812 11:24:14.236295 21885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:14.238504 21885 webserver.cc:460] Webserver started at http://127.12.110.254:36037/ using document root <none> and password file <none>
I20260812 11:24:14.239241 21885 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:14.239387 21885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:14.239701 21885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:14.242430 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "872141feaeb44653baf60be86d82a157"
format_stamp: "Formatted at 2026-08-12 11:24:14 on dist-test-slave-h6n0"
server_key: "ac148e85b7fc8ba2ae805efdf2351eb1"
server_key_iv: "85dc80948db3db6bb2f7eff508d93ab3"
server_key_version: "encryptionkey@0"
I20260812 11:24:14.243296 21885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "872141feaeb44653baf60be86d82a157"
format_stamp: "Formatted at 2026-08-12 11:24:14 on dist-test-slave-h6n0"
server_key: "ac148e85b7fc8ba2ae805efdf2351eb1"
server_key_iv: "85dc80948db3db6bb2f7eff508d93ab3"
server_key_version: "encryptionkey@0"
I20260812 11:24:14.248101 21885 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:24:14.251407 21908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:14.252677 21885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:24:14.252871 21885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "872141feaeb44653baf60be86d82a157"
format_stamp: "Formatted at 2026-08-12 11:24:14 on dist-test-slave-h6n0"
server_key: "ac148e85b7fc8ba2ae805efdf2351eb1"
server_key_iv: "85dc80948db3db6bb2f7eff508d93ab3"
server_key_version: "encryptionkey@0"
I20260812 11:24:14.253093 21885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:14.275524 21885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:14.276610 21885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:14.276958 21885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:14.299072 21885 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:43853
I20260812 11:24:14.299099 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:43853 every 8 connection(s)
I20260812 11:24:14.300441 21885 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:14.301748 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 21885
I20260812 11:24:14.302008 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:14.302918 12731 external_mini_cluster.cc:1428] Setting key 863ea4af9dd6a18884aa74d7d81f349b
I20260812 11:24:14.305225 21999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:14.318986 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:14.304941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36800 (local address 127.12.110.254:43853)
0812 11:24:14.306149 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:14.306163 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:14.306215 (+ 52us) server_negotiation.cc:407] Connection header received
0812 11:24:14.308371 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:14.308398 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:14.308973 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:14.309444 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:14.309957 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:14.312007 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:14.313385 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:14.314468 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:14.314887 (+ 419us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:14.315913 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:14.315923 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:14.315940 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:24:14.316717 (+ 777us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:14.316811 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:24:14.316848 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:14.316971 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:14.317230 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 11:24:14.317586 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":503,"threads_started":1}
I20260812 11:24:14.320782 21999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157: Bootstrap starting.
I20260812 11:24:14.323092 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:24:14.324507 21999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:14.325865 21999 log.cc:824] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:14.328810 21999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157: No bootstrap required, opened a new log
I20260812 11:24:14.336033 21999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872141feaeb44653baf60be86d82a157" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43853 } }
I20260812 11:24:14.336468 21999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:14.336589 21999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 872141feaeb44653baf60be86d82a157, State: Initialized, Role: FOLLOWER
I20260812 11:24:14.337165 21999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872141feaeb44653baf60be86d82a157" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43853 } }
I20260812 11:24:14.337414 21999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:14.337569 21999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:14.337762 21999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:14.339955 21999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872141feaeb44653baf60be86d82a157" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43853 } }
I20260812 11:24:14.340468 21999 leader_election.cc:302] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 872141feaeb44653baf60be86d82a157; no voters:
I20260812 11:24:14.341152 21999 leader_election.cc:288] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:14.341279 22007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:14.342104 22007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [term 1 LEADER]: Becoming Leader. State: Replica: 872141feaeb44653baf60be86d82a157, State: Running, Role: LEADER
I20260812 11:24:14.342751 22007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872141feaeb44653baf60be86d82a157" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43853 } }
I20260812 11:24:14.343400 21999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:14.345654 22008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "872141feaeb44653baf60be86d82a157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872141feaeb44653baf60be86d82a157" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43853 } } }
I20260812 11:24:14.346004 22008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:14.345782 22010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 872141feaeb44653baf60be86d82a157. Latest consensus state: current_term: 1 leader_uuid: "872141feaeb44653baf60be86d82a157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "872141feaeb44653baf60be86d82a157" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43853 } } }
I20260812 11:24:14.346203 22010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:14.350874 22021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:14.354933 22021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:14.363590 22021 catalog_manager.cc:1347] Generated new cluster ID: f3bcf25401c946a28ead0f5a385c3847
I20260812 11:24:14.363742 22021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:14.384096 22021 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:14.385315 22021 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:14.396867 22021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 872141feaeb44653baf60be86d82a157: Generated new TSK 0
I20260812 11:24:14.397934 22021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:24:14.476042 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:14.421965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.12.110.254:43853)
0812 11:24:14.422201 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:14.422207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:14.422589 (+ 382us) server_negotiation.cc:407] Connection header received
0812 11:24:14.422823 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:14.422828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:14.422901 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:14.423031 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0812 11:24:14.423595 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:14.424548 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:14.426926 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:14.427385 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:14.427500 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:14.428357 (+ 857us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0812 11:24:14.475444 (+ 47087us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 11:24:14.475802 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":88}
W20260812 11:24:14.476210 22003 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36816: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260812 11:24:14.476575 22047 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:14.421772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:43853 (local address 127.0.0.1:36816)
0812 11:24:14.422431 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:14.422450 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:14.422631 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:14.423081 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:14.423086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:14.423100 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0812 11:24:14.423421 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:14.423433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:14.424732 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:14.424738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:14.426763 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:14.426780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:14.427573 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:14.427580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:14.427769 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:14.427824 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0812 11:24:14.475787 (+ 47963us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0812 11:24:14.476143 (+ 356us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.12.110.254:43853: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":347,"threads_started":1}
I20260812 11:24:14.478045 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 21885
I20260812 11:24:14.479200 21992 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:14.579952 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 21885
2026-08-12T11:24:14Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (813 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (441 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (436 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:24:15 dist-test-slave-h6n0 krb5kdc[22081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:24:15 dist-test-slave-h6n0 krb5kdc[22081](info): set up 2 sockets
Aug 12 11:24:15 dist-test-slave-h6n0 krb5kdc[22081](info): commencing operation
krb5kdc: starting...
W20260812 11:24:16.861243 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.355s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:24:17 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:24:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:24:17.170248 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36499
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:17.518358 22104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:17.518764 22104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:17.519040 22104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:17.526996 22104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:17.527099 22104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:17.527204 22104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:17.527315 22104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:17.527398 22104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:17.537247 22104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:17.538604 22104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:17.539886 22104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:17.551874 22118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:17.551903 22116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:17.551892 22113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:17.553119 22104 server_base.cc:1034] running on GCE node
I20260812 11:24:17.553941 22104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:17.555361 22104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:17.556572 22104 hybrid_clock.cc:648] HybridClock initialized: now 1786533857556532 us; error 48 us; skew 500 ppm
I20260812 11:24:17.556921 22104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:17 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533857, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:17.563782 22104 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:24:17.565315 22104 webserver.cc:460] Webserver started at http://127.12.110.254:40887/ using document root <none> and password file <none>
I20260812 11:24:17.566033 22104 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:17.566216 22104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:17.566612 22104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:17.569723 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "13da05d3ad86459f97f5266e4462d954"
format_stamp: "Formatted at 2026-08-12 11:24:17 on dist-test-slave-h6n0"
server_key: "7e7f28e547f7ff7318402d516a0d6317"
server_key_iv: "33e51dbe8ead8f0ee3e43c8e3480f877"
server_key_version: "encryptionkey@0"
I20260812 11:24:17.570703 22104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "13da05d3ad86459f97f5266e4462d954"
format_stamp: "Formatted at 2026-08-12 11:24:17 on dist-test-slave-h6n0"
server_key: "7e7f28e547f7ff7318402d516a0d6317"
server_key_iv: "33e51dbe8ead8f0ee3e43c8e3480f877"
server_key_version: "encryptionkey@0"
I20260812 11:24:17.575896 22104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:24:17.579509 22132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:17.581601 22104 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:24:17.581827 22104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "13da05d3ad86459f97f5266e4462d954"
format_stamp: "Formatted at 2026-08-12 11:24:17 on dist-test-slave-h6n0"
server_key: "7e7f28e547f7ff7318402d516a0d6317"
server_key_iv: "33e51dbe8ead8f0ee3e43c8e3480f877"
server_key_version: "encryptionkey@0"
I20260812 11:24:17.582018 22104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:17.612035 22104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:17.618510 22104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:17.618909 22104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:17.642236 22104 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:36499
I20260812 11:24:17.642267 22223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:36499 every 8 connection(s)
I20260812 11:24:17.643733 22104 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:17.649015 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 22104
I20260812 11:24:17.649194 22224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:17.649458 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:17.650211 12731 external_mini_cluster.cc:1428] Setting key 545502cf6dddd559326a077b4027493d
I20260812 11:24:17.664162 22224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954: Bootstrap starting.
Aug 12 11:24:17 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:17.668514 22224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:17.670097 22224 log.cc:824] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:17.673808 22224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954: No bootstrap required, opened a new log
I20260812 11:24:17.677856 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:17.652396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.12.110.254:36499)
0812 11:24:17.653263 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:17.653278 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:17.653342 (+ 64us) server_negotiation.cc:407] Connection header received
0812 11:24:17.656113 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:17.656145 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:17.656688 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:17.657155 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:17.658502 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:17.660611 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:17.661882 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:17.662954 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:17.663127 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:17.667593 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:17.667608 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:17.667655 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:17.667740 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:17.672331 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:17.673189 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:17.673200 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:17.673212 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:17.673453 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:17.674268 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:17.674273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:17.674277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:17.675092 (+ 815us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:17.675419 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:17.675925 (+ 506us) server_negotiation.cc:299] Negotiation successful
0812 11:24:17.676416 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":361,"threads_started":1}
I20260812 11:24:17.683256 22224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da05d3ad86459f97f5266e4462d954" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36499 } }
I20260812 11:24:17.683589 22224 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:17.683668 22224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13da05d3ad86459f97f5266e4462d954, State: Initialized, Role: FOLLOWER
I20260812 11:24:17.684187 22224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da05d3ad86459f97f5266e4462d954" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36499 } }
I20260812 11:24:17.684404 22224 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:17.684501 22224 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:17.684621 22224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:17.687265 22224 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da05d3ad86459f97f5266e4462d954" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36499 } }
I20260812 11:24:17.687711 22224 leader_election.cc:302] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13da05d3ad86459f97f5266e4462d954; no voters:
I20260812 11:24:17.688273 22224 leader_election.cc:288] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:17.688436 22237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:17.689146 22237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [term 1 LEADER]: Becoming Leader. State: Replica: 13da05d3ad86459f97f5266e4462d954, State: Running, Role: LEADER
I20260812 11:24:17.689807 22237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da05d3ad86459f97f5266e4462d954" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36499 } }
I20260812 11:24:17.690521 22224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:17.692791 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13da05d3ad86459f97f5266e4462d954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da05d3ad86459f97f5266e4462d954" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36499 } } }
I20260812 11:24:17.692763 22239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13da05d3ad86459f97f5266e4462d954. Latest consensus state: current_term: 1 leader_uuid: "13da05d3ad86459f97f5266e4462d954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13da05d3ad86459f97f5266e4462d954" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36499 } } }
I20260812 11:24:17.693264 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:17.693320 22239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:17.700049 22246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:17.707288 22246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:17.718706 22246 catalog_manager.cc:1347] Generated new cluster ID: 91c1b2ca5a90422c994a93683ff6beab
I20260812 11:24:17.718855 22246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:17.728518 22246 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:17.729825 22246 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:17.742709 22246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13da05d3ad86459f97f5266e4462d954: Generated new TSK 0
I20260812 11:24:17.744267 22246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:24:17.826718 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36499
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:18.189424 22276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:18.189708 22276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:18.189872 22276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:18.197834 22276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:18.198050 22276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:18.198400 22276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:18.209064 22276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:18.210435 22276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:18.211834 22276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:18.225507 22288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:18.225591 22289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:18.225541 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:18.226905 22276 server_base.cc:1034] running on GCE node
I20260812 11:24:18.227587 22276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:18.228564 22276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:18.229815 22276 hybrid_clock.cc:648] HybridClock initialized: now 1786533858229768 us; error 54 us; skew 500 ppm
I20260812 11:24:18.230142 22276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:18 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533858, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:18.233600 22276 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:24:18.235770 22276 webserver.cc:460] Webserver started at http://127.12.110.193:35215/ using document root <none> and password file <none>
I20260812 11:24:18.236646 22276 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:18.236814 22276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:18.237160 22276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:18.240258 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "45038c4233cc45f4a9d84405f3d07ffe"
format_stamp: "Formatted at 2026-08-12 11:24:18 on dist-test-slave-h6n0"
server_key: "78fe9df3c820277a178de6a937e7fd63"
server_key_iv: "55e5903e1675968e6f098ed134bb3f1b"
server_key_version: "encryptionkey@0"
I20260812 11:24:18.241087 22276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "45038c4233cc45f4a9d84405f3d07ffe"
format_stamp: "Formatted at 2026-08-12 11:24:18 on dist-test-slave-h6n0"
server_key: "78fe9df3c820277a178de6a937e7fd63"
server_key_iv: "55e5903e1675968e6f098ed134bb3f1b"
server_key_version: "encryptionkey@0"
I20260812 11:24:18.246745 22276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 11:24:18.250267 22302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:18.251758 22276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:24:18.251964 22276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "45038c4233cc45f4a9d84405f3d07ffe"
format_stamp: "Formatted at 2026-08-12 11:24:18 on dist-test-slave-h6n0"
server_key: "78fe9df3c820277a178de6a937e7fd63"
server_key_iv: "55e5903e1675968e6f098ed134bb3f1b"
server_key_version: "encryptionkey@0"
I20260812 11:24:18.252187 22276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:18.288412 22276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:18.293349 22276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:18.293670 22276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:18.294780 22276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:18.296183 22276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:18.296293 22276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:18.296458 22276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:18.296543 22276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:18.337137 22276 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:39173
I20260812 11:24:18.337201 22483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:39173 every 8 connection(s)
I20260812 11:24:18.338502 22276 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:18.340123 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 22276
I20260812 11:24:18.340371 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:24:18.341288 12731 external_mini_cluster.cc:1428] Setting key 52d4b7d9e20a0d503da7cc831dcdd749
Aug 12 11:24:18 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533858, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:18.363926 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:18.341454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:55111 (local address 127.12.110.254:36499)
0812 11:24:18.341926 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:18.341931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:18.343127 (+ 1196us) server_negotiation.cc:407] Connection header received
0812 11:24:18.345955 (+ 2828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:18.345962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:18.346101 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:18.346455 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:18.349550 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.350463 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.352266 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.352676 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.352823 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:18.357385 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:18.357390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:18.357394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:18.357428 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:18.360481 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:18.361372 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:18.361377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:18.361380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:18.361512 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:18.362009 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:18.362019 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:18.362021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:18.362301 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:18.362468 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:18.363292 (+ 824us) server_negotiation.cc:299] Negotiation successful
0812 11:24:18.363535 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260812 11:24:18.365432 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:18.342093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36499 (local address 127.12.110.193:55111)
0812 11:24:18.342956 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:18.343016 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:18.345424 (+ 2408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:18.346862 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:18.346874 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:18.348018 (+ 1144us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:18.349129 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:18.349254 (+ 125us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.350722 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.350727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:18.352092 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:18.352106 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.352935 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.352943 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:18.353195 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:18.354041 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:18.354065 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:18.357085 (+ 3020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:18.360672 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:18.360683 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:18.360702 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:18.361226 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:18.361644 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:18.361649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:18.361653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:18.361867 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:18.362442 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:18.362451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:18.362881 (+ 430us) client_negotiation.cc:769] Sending connection context
0812 11:24:18.363178 (+ 297us) client_negotiation.cc:240] Negotiation successful
0812 11:24:18.363737 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":426,"threads_started":1}
I20260812 11:24:18.368234 22484 heartbeater.cc:347] Connected to a master server at 127.12.110.254:36499
I20260812 11:24:18.368795 22484 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:18.369887 22484 heartbeater.cc:511] Master 127.12.110.254:36499 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
I20260812 11:24:18.372776 22164 ts_manager.cc:194] Registered new tserver with Master: 45038c4233cc45f4a9d84405f3d07ffe (127.12.110.193:39173)
I20260812 11:24:18.374464 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:55111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:24:18.410077 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36499
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:18.765487 22491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:18.765813 22491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:18.765961 22491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:18.774397 22491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:18.774597 22491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:18.774773 22491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:18.785156 22491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:18.786640 22491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:18.788414 22491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:18.804512 22496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:18.805024 22500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:18.804489 22498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:18.806629 22491 server_base.cc:1034] running on GCE node
I20260812 11:24:18.807443 22491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:18.808660 22491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:18.810051 22491 hybrid_clock.cc:648] HybridClock initialized: now 1786533858809972 us; error 57 us; skew 500 ppm
I20260812 11:24:18.810865 22491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:18 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533858, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:18.815039 22491 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:24:18.816712 22491 webserver.cc:460] Webserver started at http://127.12.110.194:35059/ using document root <none> and password file <none>
I20260812 11:24:18.817425 22491 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:18.817533 22491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:18.817790 22491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:18.821295 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "a956048d7374408fa015e2f16906ef86"
format_stamp: "Formatted at 2026-08-12 11:24:18 on dist-test-slave-h6n0"
server_key: "b9ff4dce6ae1fd78b4b4309085851f5d"
server_key_iv: "51373ec7377d7353bc6c7b1128bd4cbf"
server_key_version: "encryptionkey@0"
I20260812 11:24:18.822070 22491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "a956048d7374408fa015e2f16906ef86"
format_stamp: "Formatted at 2026-08-12 11:24:18 on dist-test-slave-h6n0"
server_key: "b9ff4dce6ae1fd78b4b4309085851f5d"
server_key_iv: "51373ec7377d7353bc6c7b1128bd4cbf"
server_key_version: "encryptionkey@0"
I20260812 11:24:18.827534 22491 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 11:24:18.832012 22510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:18.833787 22491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260812 11:24:18.834002 22491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "a956048d7374408fa015e2f16906ef86"
format_stamp: "Formatted at 2026-08-12 11:24:18 on dist-test-slave-h6n0"
server_key: "b9ff4dce6ae1fd78b4b4309085851f5d"
server_key_iv: "51373ec7377d7353bc6c7b1128bd4cbf"
server_key_version: "encryptionkey@0"
I20260812 11:24:18.834234 22491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:18.854030 22491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:18.859699 22491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:18.860018 22491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:18.861068 22491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:18.862494 22491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:18.862637 22491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 11:24:18.862783 22491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:18.862844 22491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:18.904742 22491 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:34261
I20260812 11:24:18.904781 22683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:34261 every 8 connection(s)
I20260812 11:24:18.906145 22491 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:18.911458 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 22491
I20260812 11:24:18.911706 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:24:18.912249 12731 external_mini_cluster.cc:1428] Setting key 93d567e440cbd7529e9e1abaafaf3577
Aug 12 11:24:18 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533858, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:18.933480 22686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:18.909501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:37277 (local address 127.12.110.254:36499)
0812 11:24:18.910234 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:18.910239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:18.911930 (+ 1691us) server_negotiation.cc:407] Connection header received
0812 11:24:18.914287 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:18.914292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:18.914380 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:18.914520 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:18.916183 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.917399 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.919158 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.919902 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.920055 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:18.925513 (+ 5458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:18.925524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:18.925532 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:18.925598 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:18.929756 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:18.930911 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:18.930916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:18.930918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:18.931075 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:18.931627 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:18.931630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:18.931632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:18.931908 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:18.932025 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:18.932993 (+ 968us) server_negotiation.cc:299] Negotiation successful
0812 11:24:18.933184 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":391,"threads_started":1}
I20260812 11:24:18.935424 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:18.910481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36499 (local address 127.12.110.194:37277)
0812 11:24:18.911705 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:18.911769 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:18.913967 (+ 2198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:18.914791 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:18.914800 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:18.915364 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:18.915971 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:18.915989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.917601 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.917605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:18.918991 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:18.919005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:18.920327 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:18.920335 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:18.920748 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:18.921633 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:18.921682 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:18.925063 (+ 3381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:18.929950 (+ 4887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:18.929962 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:18.929985 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:18.930743 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:18.931218 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:18.931226 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:18.931233 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:18.931507 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:18.932042 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:18.932053 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:18.932559 (+ 506us) client_negotiation.cc:769] Sending connection context
0812 11:24:18.932875 (+ 316us) client_negotiation.cc:240] Negotiation successful
0812 11:24:18.933554 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"thread_start_us":558,"threads_started":1}
I20260812 11:24:18.937165 22684 heartbeater.cc:347] Connected to a master server at 127.12.110.254:36499
I20260812 11:24:18.937532 22684 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:18.938486 22684 heartbeater.cc:511] Master 127.12.110.254:36499 requested a full tablet report, sending...
I20260812 11:24:18.940410 22164 ts_manager.cc:194] Registered new tserver with Master: a956048d7374408fa015e2f16906ef86 (127.12.110.194:34261)
I20260812 11:24:18.941268 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:37277
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:24:18.986143 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36499
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:39379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:19.335453 22690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:19.335754 22690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:19.335851 22690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:19.343580 22690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:19.343696 22690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:19.343811 22690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:19.353577 22690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:39379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:36499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:19.354902 22690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:19.356204 22690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:19.369437 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:19.369413 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:19.369508 22704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:19.370658 22690 server_base.cc:1034] running on GCE node
I20260812 11:24:19.371348 22690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:19.372291 22690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:19.373538 22690 hybrid_clock.cc:648] HybridClock initialized: now 1786533859373489 us; error 44 us; skew 500 ppm
I20260812 11:24:19.373855 22690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.377425 22690 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:24:19.378105 22484 heartbeater.cc:503] Master 127.12.110.254:36499 was elected leader, sending a full tablet report...
I20260812 11:24:19.379383 22690 webserver.cc:460] Webserver started at http://127.12.110.195:35501/ using document root <none> and password file <none>
I20260812 11:24:19.380215 22690 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:19.380388 22690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:19.380728 22690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:19.383986 22690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "41ebd508a5d042ce8e468c45ae88fe32"
format_stamp: "Formatted at 2026-08-12 11:24:19 on dist-test-slave-h6n0"
server_key: "b62ac607b05079dc72c98dc2fc06ddd4"
server_key_iv: "13fd017f49d8275bdcc47dd4b2fd1149"
server_key_version: "encryptionkey@0"
I20260812 11:24:19.384904 22690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "41ebd508a5d042ce8e468c45ae88fe32"
format_stamp: "Formatted at 2026-08-12 11:24:19 on dist-test-slave-h6n0"
server_key: "b62ac607b05079dc72c98dc2fc06ddd4"
server_key_iv: "13fd017f49d8275bdcc47dd4b2fd1149"
server_key_version: "encryptionkey@0"
I20260812 11:24:19.390905 22690 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260812 11:24:19.394794 22716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:19.396206 22690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:24:19.396404 22690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "41ebd508a5d042ce8e468c45ae88fe32"
format_stamp: "Formatted at 2026-08-12 11:24:19 on dist-test-slave-h6n0"
server_key: "b62ac607b05079dc72c98dc2fc06ddd4"
server_key_iv: "13fd017f49d8275bdcc47dd4b2fd1149"
server_key_version: "encryptionkey@0"
I20260812 11:24:19.396633 22690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:19.429610 22690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:19.434351 22690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:19.434715 22690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:19.435765 22690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:19.437251 22690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:19.437392 22690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:19.437615 22690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:19.437701 22690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:19.472177 22690 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:43915
I20260812 11:24:19.472218 22916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:43915 every 8 connection(s)
I20260812 11:24:19.473517 22690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:19.477329 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 22690
I20260812 11:24:19.477523 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:24:19.478068 12731 external_mini_cluster.cc:1428] Setting key 9c00ec2d9a7a53f658e3a7e8d62cf7fe
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:19.496271 22920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.476264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:35413 (local address 127.12.110.254:36499)
0812 11:24:19.476730 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.476735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.477917 (+ 1182us) server_negotiation.cc:407] Connection header received
0812 11:24:19.480600 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.480604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.480693 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.480788 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.482433 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.483294 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.484944 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.485524 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.485634 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.489219 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:19.489224 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:19.489228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:19.489261 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:19.492708 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.493657 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.493664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.493669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.493807 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.494306 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.494311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.494327 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.494685 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:19.494869 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:19.495610 (+ 741us) server_negotiation.cc:299] Negotiation successful
0812 11:24:19.495851 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":234,"threads_started":1}
I20260812 11:24:19.497265 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.476802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36499 (local address 127.12.110.195:35413)
0812 11:24:19.477748 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:19.477793 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:19.480286 (+ 2493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:19.481062 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:19.481074 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:19.481693 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:19.482236 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:19.482254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.483517 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.483521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:19.484785 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:19.484798 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.485759 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.485765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:19.485980 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.486816 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:19.486838 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:19.489020 (+ 2182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:19.492873 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:19.492884 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:19.492900 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:19.493528 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:19.493955 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:19.493961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:19.493965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:19.494177 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:19.494837 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:19.494846 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:19.495202 (+ 356us) client_negotiation.cc:769] Sending connection context
0812 11:24:19.495507 (+ 305us) client_negotiation.cc:240] Negotiation successful
0812 11:24:19.496074 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":382,"threads_started":1}
I20260812 11:24:19.498781 22918 heartbeater.cc:347] Connected to a master server at 127.12.110.254:36499
I20260812 11:24:19.499109 22918 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:19.499835 22918 heartbeater.cc:511] Master 127.12.110.254:36499 requested a full tablet report, sending...
I20260812 11:24:19.501381 22164 ts_manager.cc:194] Registered new tserver with Master: 41ebd508a5d042ce8e468c45ae88fe32 (127.12.110.195:43915)
I20260812 11:24:19.502220 22164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:35413
I20260812 11:24:19.506433 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/test-admin.keytab.
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.540866 12731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260812 11:24:19.547062 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:19.544237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.12.110.193:39173)
0812 11:24:19.544944 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.544951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.544978 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:24:19.545044 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.545053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.545278 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.545412 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.546512 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38728: BlockingRecv error: recv got EOF from 127.0.0.1:38728 (error 108)
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":390,"threads_started":1}
W20260812 11:24:19.547047 12731 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.549050 12731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.550951 12731 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:24:19.569271 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.552936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38744 (local address 127.12.110.193:39173)
0812 11:24:19.553175 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.553182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.553445 (+ 263us) server_negotiation.cc:407] Connection header received
0812 11:24:19.553661 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.553666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.553741 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.553849 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.554947 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.556018 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.557302 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.557893 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.558046 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.563008 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:19.563021 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:19.563038 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:19.563075 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:19.566227 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.567092 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.567099 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.567102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.567225 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.567637 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.567641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.567644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.568010 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:19.568351 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:19.568693 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 11:24:19.568971 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260812 11:24:19.576661 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:19.574120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.12.110.193:39173)
0812 11:24:19.574279 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.574282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.574717 (+ 435us) server_negotiation.cc:407] Connection header received
0812 11:24:19.574828 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.574832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.574892 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.574972 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.576337 (+ 1365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38748: BlockingRecv error: recv got EOF from 127.0.0.1:38748 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 11:24:19.576694 12731 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.578508 12731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.580158 12731 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:24:19.596578 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.581504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38764 (local address 127.12.110.193:39173)
0812 11:24:19.581774 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.581783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.582091 (+ 308us) server_negotiation.cc:407] Connection header received
0812 11:24:19.582283 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.582289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.582349 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.582456 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.583521 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.584235 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.585432 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.585936 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.586075 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.589827 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:19.589832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:19.589835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:19.589877 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:19.593045 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.594167 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.594171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.594173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.594310 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.594962 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.594969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.594971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.595392 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:19.595781 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:19.596010 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 11:24:19.596250 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
W20260812 11:24:19.602005 22930 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:19.599690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.12.110.193:39173)
0812 11:24:19.600227 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.600234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.600343 (+ 109us) server_negotiation.cc:407] Connection header received
0812 11:24:19.600566 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.600574 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.600674 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.600786 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.601720 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38778: BlockingRecv error: recv got EOF from 127.0.0.1:38778 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260812 11:24:19.602083 12731 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786533802845873-12731-0/krb5kdc/krb5cc' of type FILE
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.603747 12731 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:19.605683 12731 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:24:19.622129 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.607103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.12.110.193:39173)
0812 11:24:19.607267 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.607271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.607724 (+ 453us) server_negotiation.cc:407] Connection header received
0812 11:24:19.607925 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.607932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.608009 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.608139 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.609102 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.609822 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.611140 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.611571 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.611701 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.615457 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:19.615464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:19.615469 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:19.615517 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:19.618667 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.619653 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.619661 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.619665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.619940 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.620505 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.620512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.620516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.620852 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:19.621236 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:19.621522 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 11:24:19.621759 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:24:19.624087 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 22276
I20260812 11:24:19.637012 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.624945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.12.110.193:39173)
0812 11:24:19.625195 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.625202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.625540 (+ 338us) server_negotiation.cc:407] Connection header received
0812 11:24:19.625736 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.625741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.625810 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.625920 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.626910 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.627591 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.628799 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.629235 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.629383 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.631539 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:19.631548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:19.631557 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:19.631587 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:19.634112 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.634925 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.634931 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.634935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.635107 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.635519 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.635532 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.635535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.635812 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:19.636131 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:19.636451 (+ 320us) server_negotiation.cc:299] Negotiation successful
0812 11:24:19.636674 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260812 11:24:19.637777 22478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:19.725930 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 22276
I20260812 11:24:19.757016 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 22491
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:24:19.775475 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.758060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.12.110.194:34261)
0812 11:24:19.758690 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.758697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.758722 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:24:19.758790 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.758796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.758998 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.759135 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.760686 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.762140 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.763646 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.764242 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.764372 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.768644 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:19.768657 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:19.768671 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:19.768711 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:19.772136 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.772950 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.772961 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.772964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.773322 (+ 358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.773771 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.773775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.773778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.774197 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:19.774458 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:19.774869 (+ 411us) server_negotiation.cc:299] Negotiation successful
0812 11:24:19.775132 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":325,"threads_started":1}
I20260812 11:24:19.776769 22675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:19.862679 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 22491
I20260812 11:24:19.888553 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 22690
Aug 12 11:24:19 dist-test-slave-h6n0 krb5kdc[22081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533859, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:24:19.907222 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:19.889747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45718 (local address 127.12.110.195:43915)
0812 11:24:19.890763 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:19.890812 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:19.890856 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:24:19.890923 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:19.890930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:19.891157 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:19.891351 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:19.892693 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.893701 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.895142 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:19.895699 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:19.895849 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:19.900037 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:19.900049 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:19.900100 (+ 51us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:19.900149 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:19.903485 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.905027 (+ 1542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.905034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.905037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.905175 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:19.905674 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:19.905677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:19.905679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:19.906063 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:19.906298 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:19.906706 (+ 408us) server_negotiation.cc:299] Negotiation successful
0812 11:24:19.906914 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":569,"threads_started":1}
I20260812 11:24:19.908526 22908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:19.995193 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 22690
I20260812 11:24:20.020658 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 22104
I20260812 11:24:20.021831 22215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:20.128983 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 22104
2026-08-12T11:24:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4677 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:24:20 dist-test-slave-h6n0 krb5kdc[22968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:24:20 dist-test-slave-h6n0 krb5kdc[22968](info): set up 2 sockets
Aug 12 11:24:20 dist-test-slave-h6n0 krb5kdc[22968](info): commencing operation
krb5kdc: starting...
W20260812 11:24:21.537286 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.353s user 0.004s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:24:21 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab.
Entry for principal oryx/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab.
I20260812 11:24:21.829725 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:44299
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab
--principal=oryx/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:22.189244 22988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:22.189539 22988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:22.189718 22988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:22.197780 22988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:22.197896 22988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:22.198011 22988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:22.198148 22988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:22.198230 22988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:22.208284 22988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:44299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab
--principal=oryx/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:22.209673 22988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:22.211030 22988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:22.224534 23000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:22.224800 22995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:22.224800 22996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:22.225272 22988 server_base.cc:1034] running on GCE node
I20260812 11:24:22.226351 22988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:22.227870 22988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:22.229075 22988 hybrid_clock.cc:648] HybridClock initialized: now 1786533862229037 us; error 42 us; skew 500 ppm
I20260812 11:24:22.229391 22988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:22 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533862, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:22.233691 22988 init.cc:377] Logged in from keytab as oryx/127.12.110.254@KRBTEST.COM (short username oryx)
I20260812 11:24:22.235800 22988 webserver.cc:460] Webserver started at http://127.12.110.254:37573/ using document root <none> and password file <none>
I20260812 11:24:22.236472 22988 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:22.236580 22988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:22.236824 22988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:22.239887 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "5cde5ff1cbff4d0f8274ef992177d97e"
format_stamp: "Formatted at 2026-08-12 11:24:22 on dist-test-slave-h6n0"
server_key: "9d0ab464ba93a07ee97d71929ade67f0"
server_key_iv: "cd14666cb9d4b2393022dacca2964f64"
server_key_version: "encryptionkey@0"
I20260812 11:24:22.240593 22988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "5cde5ff1cbff4d0f8274ef992177d97e"
format_stamp: "Formatted at 2026-08-12 11:24:22 on dist-test-slave-h6n0"
server_key: "9d0ab464ba93a07ee97d71929ade67f0"
server_key_iv: "cd14666cb9d4b2393022dacca2964f64"
server_key_version: "encryptionkey@0"
I20260812 11:24:22.245831 22988 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 11:24:22.249279 23013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:22.250800 22988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:24:22.251003 22988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "5cde5ff1cbff4d0f8274ef992177d97e"
format_stamp: "Formatted at 2026-08-12 11:24:22 on dist-test-slave-h6n0"
server_key: "9d0ab464ba93a07ee97d71929ade67f0"
server_key_iv: "cd14666cb9d4b2393022dacca2964f64"
server_key_version: "encryptionkey@0"
I20260812 11:24:22.251186 22988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:22.267839 22988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:22.273039 22988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:22.273406 22988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:22.294487 22988 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:44299
I20260812 11:24:22.294584 23092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:44299 every 8 connection(s)
I20260812 11:24:22.295926 22988 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:22.300395 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 22988
I20260812 11:24:22.300616 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:22.300869 23093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:22.301358 12731 external_mini_cluster.cc:1428] Setting key b7209e4e90b98a54c3575bb8b0f44dda
Aug 12 11:24:22 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:22.316849 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: Bootstrap starting.
I20260812 11:24:22.321350 23093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:22.322824 23093 log.cc:824] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:22.326428 23093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: No bootstrap required, opened a new log
I20260812 11:24:22.326982 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:22.303313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.12.110.254:44299)
0812 11:24:22.304276 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:22.304298 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:22.304351 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:24:22.306376 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:22.306405 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:22.306988 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:22.307425 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:22.308726 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:22.310990 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:22.312265 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:22.313472 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:22.313671 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:22.317680 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:22.317691 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:22.317709 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:22.317763 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:22.322277 (+ 4514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:22.323036 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:22.323044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:22.323053 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:22.323279 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:22.323782 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:22.323786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:22.323788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:22.324491 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:22.324785 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:22.325185 (+ 400us) server_negotiation.cc:299] Negotiation successful
0812 11:24:22.325632 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":401,"threads_started":1}
I20260812 11:24:22.334816 23093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } }
I20260812 11:24:22.335143 23093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:22.335215 23093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cde5ff1cbff4d0f8274ef992177d97e, State: Initialized, Role: FOLLOWER
I20260812 11:24:22.335891 23093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } }
I20260812 11:24:22.336199 23093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:22.336371 23093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:22.336546 23093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:22.338790 23093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } }
I20260812 11:24:22.339274 23093 leader_election.cc:302] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cde5ff1cbff4d0f8274ef992177d97e; no voters:
I20260812 11:24:22.339871 23093 leader_election.cc:288] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:22.340000 23099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:22.340739 23099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 1 LEADER]: Becoming Leader. State: Replica: 5cde5ff1cbff4d0f8274ef992177d97e, State: Running, Role: LEADER
I20260812 11:24:22.341266 23099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } }
I20260812 11:24:22.342005 23093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:22.344120 23101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } }
I20260812 11:24:22.344452 23101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:22.344925 23105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde5ff1cbff4d0f8274ef992177d97e. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } }
I20260812 11:24:22.345254 23105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:22.346454 23112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:22.352320 23112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:22.361310 23112 catalog_manager.cc:1347] Generated new cluster ID: 490a3b38963f4c17bd8d048ac6a3c817
I20260812 11:24:22.361483 23112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:22.402331 23112 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:22.403651 23112 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:22.410303 23112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: Generated new TSK 0
I20260812 11:24:22.411391 23112 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab.
Entry for principal oryx/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab.
I20260812 11:24:22.490955 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44299
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab
--principal=oryx/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:22.874676 23136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:22.874974 23136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:22.875129 23136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:22.884637 23136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:22.884769 23136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:22.884948 23136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:22.895718 23136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab
--principal=oryx/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:22.897069 23136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:22.898366 23136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:22.912001 23145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:22.912030 23146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:22.912187 23148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:22.913292 23136 server_base.cc:1034] running on GCE node
I20260812 11:24:22.913913 23136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:22.914908 23136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:22.916124 23136 hybrid_clock.cc:648] HybridClock initialized: now 1786533862916067 us; error 53 us; skew 500 ppm
I20260812 11:24:22.916466 23136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:22 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533862, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:22.925130 23136 init.cc:377] Logged in from keytab as oryx/127.12.110.193@KRBTEST.COM (short username oryx)
I20260812 11:24:22.926746 23136 webserver.cc:460] Webserver started at http://127.12.110.193:37183/ using document root <none> and password file <none>
I20260812 11:24:22.927443 23136 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:22.927585 23136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:22.927925 23136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:22.930950 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "bc8713ea524e4655b599a1619c945cc4"
format_stamp: "Formatted at 2026-08-12 11:24:22 on dist-test-slave-h6n0"
server_key: "daff1e624944caf5f309481522fe3c4f"
server_key_iv: "3572f28f94b12e9f5d260da261a39379"
server_key_version: "encryptionkey@0"
I20260812 11:24:22.931764 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "bc8713ea524e4655b599a1619c945cc4"
format_stamp: "Formatted at 2026-08-12 11:24:22 on dist-test-slave-h6n0"
server_key: "daff1e624944caf5f309481522fe3c4f"
server_key_iv: "3572f28f94b12e9f5d260da261a39379"
server_key_version: "encryptionkey@0"
I20260812 11:24:22.936801 23136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 11:24:22.940218 23159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:22.941597 23136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:24:22.941737 23136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "bc8713ea524e4655b599a1619c945cc4"
format_stamp: "Formatted at 2026-08-12 11:24:22 on dist-test-slave-h6n0"
server_key: "daff1e624944caf5f309481522fe3c4f"
server_key_iv: "3572f28f94b12e9f5d260da261a39379"
server_key_version: "encryptionkey@0"
I20260812 11:24:22.941964 23136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:22.961833 23136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:22.966266 23136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:22.966591 23136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:22.967587 23136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:22.968942 23136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:22.969020 23136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:22.969152 23136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:22.969218 23136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:23.006102 23136 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:46721
I20260812 11:24:23.006145 23354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:46721 every 8 connection(s)
I20260812 11:24:23.007460 23136 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:23.012224 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 23136
I20260812 11:24:23.012477 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:24:23.013406 12731 external_mini_cluster.cc:1428] Setting key f0d53448636ee0dfd923623f08d41665
Aug 12 11:24:23 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533862, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:23.032378 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:23.010111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:57077 (local address 127.12.110.254:44299)
0812 11:24:23.010663 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:23.010669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:23.011616 (+ 947us) server_negotiation.cc:407] Connection header received
0812 11:24:23.014429 (+ 2813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:23.014436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:23.014583 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:23.014722 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:23.016695 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.017406 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.020197 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.020574 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.020675 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:23.025506 (+ 4831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:23.025511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:23.025514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:23.025545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:23.028892 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:23.029646 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:23.029654 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:23.029660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:23.029811 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:23.030252 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:23.030260 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:23.030266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:23.030666 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:23.030818 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:23.031587 (+ 769us) server_negotiation.cc:299] Negotiation successful
0812 11:24:23.031858 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":422,"thread_start_us":276,"threads_started":1}
I20260812 11:24:23.032976 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:23.010616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:57077)
0812 11:24:23.011457 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:23.011497 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:23.014141 (+ 2644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:23.015001 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:23.015009 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:23.015653 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:23.016495 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:23.016527 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.017583 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.017593 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:23.020018 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:23.020038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.020816 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.020826 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:23.021178 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:23.021977 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:23.022023 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:23.025273 (+ 3250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:23.029044 (+ 3771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:23.029051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:23.029062 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:23.029507 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:23.029938 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:23.029941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:23.029944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:23.030128 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:23.030823 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:23.030829 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:23.031147 (+ 318us) client_negotiation.cc:769] Sending connection context
0812 11:24:23.031416 (+ 269us) client_negotiation.cc:240] Negotiation successful
0812 11:24:23.031837 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":352,"threads_started":1}
I20260812 11:24:23.034659 23357 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44299
I20260812 11:24:23.035050 23357 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:23.035928 23357 heartbeater.cc:511] Master 127.12.110.254:44299 requested a full tablet report, sending...
I20260812 11:24:23.038295 23037 ts_manager.cc:194] Registered new tserver with Master: bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:23.040613 23037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:57077
WARNING: no policy specified for oryx/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab.
Entry for principal oryx/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab.
I20260812 11:24:23.079224 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44299
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab
--principal=oryx/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:23.418089 23366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:23.418457 23366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:23.418653 23366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:23.426410 23366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:23.426692 23366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:23.427091 23366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:23.438158 23366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab
--principal=oryx/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:23.439570 23366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:23.440912 23366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:23.454898 23376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:23.454849 23382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:23.454964 23377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:23.456607 23366 server_base.cc:1034] running on GCE node
I20260812 11:24:23.457229 23366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:23.458158 23366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:23.459384 23366 hybrid_clock.cc:648] HybridClock initialized: now 1786533863459353 us; error 54 us; skew 500 ppm
I20260812 11:24:23.459707 23366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:23 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533863, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:23.462921 23366 init.cc:377] Logged in from keytab as oryx/127.12.110.194@KRBTEST.COM (short username oryx)
I20260812 11:24:23.464273 23366 webserver.cc:460] Webserver started at http://127.12.110.194:34473/ using document root <none> and password file <none>
I20260812 11:24:23.464969 23366 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:23.465111 23366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:23.465400 23366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:23.468333 23366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "b4470d8736a04cd4b16b8ffda7aba8e5"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "02939659fc0423a8f6a381f0eb35bfea"
server_key_iv: "87c8ade1a9748a05e1833ca39235dd2f"
server_key_version: "encryptionkey@0"
I20260812 11:24:23.469112 23366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "b4470d8736a04cd4b16b8ffda7aba8e5"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "02939659fc0423a8f6a381f0eb35bfea"
server_key_iv: "87c8ade1a9748a05e1833ca39235dd2f"
server_key_version: "encryptionkey@0"
I20260812 11:24:23.474921 23366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:24:23.478711 23392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:23.480865 23366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:24:23.481072 23366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "b4470d8736a04cd4b16b8ffda7aba8e5"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "02939659fc0423a8f6a381f0eb35bfea"
server_key_iv: "87c8ade1a9748a05e1833ca39235dd2f"
server_key_version: "encryptionkey@0"
I20260812 11:24:23.481302 23366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:23.496737 23366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:23.501129 23366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:23.501437 23366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:23.502395 23366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:23.503796 23366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:23.503903 23366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:23.504065 23366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:23.504138 23366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:23.538242 23366 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:41603
I20260812 11:24:23.538293 23574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:41603 every 8 connection(s)
I20260812 11:24:23.539570 23366 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:23.546372 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 23366
I20260812 11:24:23.546649 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:24:23.547286 12731 external_mini_cluster.cc:1428] Setting key 28b9bc73d62e0982dc89abdac11f95c0
Aug 12 11:24:23 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533863, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:23.563045 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:23.542696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:57007 (local address 127.12.110.254:44299)
0812 11:24:23.543293 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:23.543298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:23.544102 (+ 804us) server_negotiation.cc:407] Connection header received
0812 11:24:23.546701 (+ 2599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:23.546706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:23.546799 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:23.546932 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:23.548767 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.549725 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.551319 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.551725 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.552005 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:23.555965 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:23.555974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:23.555982 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:23.556029 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:23.559744 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:23.560492 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:23.560500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:23.560506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:23.560664 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:23.561073 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:23.561080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:23.561085 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:23.561456 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:23.561578 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:23.562519 (+ 941us) server_negotiation.cc:299] Negotiation successful
0812 11:24:23.562748 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":370,"threads_started":1}
I20260812 11:24:23.564453 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:23.543214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.194:57007)
0812 11:24:23.543936 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:23.543976 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:23.545714 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:23.547336 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:23.547343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:23.547937 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:23.548597 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:23.548615 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.549891 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.549895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:23.551177 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:23.551189 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:23.552088 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:23.552098 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:23.552338 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:23.553090 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:23.553117 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:23.555698 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:23.559883 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:23.559890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:23.559912 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:23.560359 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:23.560774 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:23.560777 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:23.560779 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:23.560953 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:23.561591 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:23.561602 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:23.562084 (+ 482us) client_negotiation.cc:769] Sending connection context
0812 11:24:23.562406 (+ 322us) client_negotiation.cc:240] Negotiation successful
0812 11:24:23.563027 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":276,"threads_started":1}
I20260812 11:24:23.565994 23576 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44299
I20260812 11:24:23.566298 23576 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:23.567112 23576 heartbeater.cc:511] Master 127.12.110.254:44299 requested a full tablet report, sending...
I20260812 11:24:23.569008 23037 ts_manager.cc:194] Registered new tserver with Master: b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603)
I20260812 11:24:23.569861 23037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:57007
WARNING: no policy specified for oryx/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab.
Entry for principal oryx/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab.
I20260812 11:24:23.613269 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44299
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab
--principal=oryx/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:23.948177 23583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:23.948444 23583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:23.948606 23583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:23.955910 23583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:23.956077 23583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:23.956290 23583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:23.966785 23583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab
--principal=oryx/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:44299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:23.968212 23583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:23.969563 23583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:23.987985 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:23.987962 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:23.988528 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:23.988791 23583 server_base.cc:1034] running on GCE node
I20260812 11:24:23.989614 23583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:23.990878 23583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:23.992405 23583 hybrid_clock.cc:648] HybridClock initialized: now 1786533863992323 us; error 95 us; skew 500 ppm
I20260812 11:24:23.992846 23583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:23 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533863, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:23.997133 23583 init.cc:377] Logged in from keytab as oryx/127.12.110.195@KRBTEST.COM (short username oryx)
I20260812 11:24:23.998916 23583 webserver.cc:460] Webserver started at http://127.12.110.195:36327/ using document root <none> and password file <none>
I20260812 11:24:23.999656 23583 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:23.999818 23583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:24.000242 23583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:24.003832 23583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "7f70661a5b854a4b85823ad896d6ab12"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "d0c51f923e4b3a3bf344bf8706181428"
server_key_iv: "675ff0ee4989e6cb5a128971f122c820"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.004684 23583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "7f70661a5b854a4b85823ad896d6ab12"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "d0c51f923e4b3a3bf344bf8706181428"
server_key_iv: "675ff0ee4989e6cb5a128971f122c820"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.011685 23583 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260812 11:24:24.015599 23604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:24.017190 23583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:24:24.017331 23583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "7f70661a5b854a4b85823ad896d6ab12"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "d0c51f923e4b3a3bf344bf8706181428"
server_key_iv: "675ff0ee4989e6cb5a128971f122c820"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.017570 23583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:24.037449 23583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:24.042840 23583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:24.043155 23583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:24.044255 23583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:24.044827 23357 heartbeater.cc:503] Master 127.12.110.254:44299 was elected leader, sending a full tablet report...
I20260812 11:24:24.045969 23583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:24.046058 23583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:24.046168 23583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:24.046232 23583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:24.082983 23583 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:45185
I20260812 11:24:24.083041 23778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:45185 every 8 connection(s)
I20260812 11:24:24.084713 23583 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:24.090444 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 23583
I20260812 11:24:24.090672 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:24:24.091189 12731 external_mini_cluster.cc:1428] Setting key faef35b814611011d96e95ad2c323e02
Aug 12 11:24:24 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533863, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:24.109826 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.087997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:46305 (local address 127.12.110.254:44299)
0812 11:24:24.088571 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.088579 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.089564 (+ 985us) server_negotiation.cc:407] Connection header received
0812 11:24:24.092490 (+ 2926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.092497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.092610 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.092700 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.095099 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.095882 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.098403 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.098890 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.098994 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.103257 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.103262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.103265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.103297 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.106753 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.107477 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.107483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.107487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.107616 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.108077 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.108082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.108085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.108375 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.108542 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.109215 (+ 673us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.109448 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":229,"threads_started":1}
I20260812 11:24:24.110852 23783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.088513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.195:46305)
0812 11:24:24.089373 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:24.089437 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:24.092154 (+ 2717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:24.093232 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:24.093244 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:24.094069 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:24.094904 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.094929 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.096052 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.096061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.098241 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.098257 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.099118 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.099126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.099447 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.100217 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:24.100251 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:24.103048 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:24.106899 (+ 3851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.106908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.106923 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.107345 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.107734 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.107739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.107743 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.107941 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.108512 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:24.108520 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:24.108860 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 11:24:24.109120 (+ 260us) client_negotiation.cc:240] Negotiation successful
0812 11:24:24.109641 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":316,"threads_started":1}
I20260812 11:24:24.112381 23780 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44299
I20260812 11:24:24.112705 23780 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:24.113436 23780 heartbeater.cc:511] Master 127.12.110.254:44299 requested a full tablet report, sending...
I20260812 11:24:24.115025 23037 ts_manager.cc:194] Registered new tserver with Master: 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185)
I20260812 11:24:24.116220 23037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:46305
I20260812 11:24:24.120537 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.110.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.253.keytab.
Entry for principal oryx/127.12.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.253.keytab.
I20260812 11:24:24.183966 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:37647
--webserver_interface=127.12.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.253.keytab
--principal=oryx/127.12.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:44299,127.12.110.253:37647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:24.526940 23786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:24.527215 23786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:24.527305 23786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:24.534519 23786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:24.534703 23786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:24.534758 23786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:24.534817 23786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:24.534855 23786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:24.544672 23786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:44299,127.12.110.253:37647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.253.keytab
--principal=oryx/127.12.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
W20260812 11:24:24.545097 23786 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.12.110.254:44299,127.12.110.253:37647'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260812 11:24:24.546195 23786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:24.547655 23786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:24.559999 23794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:24.560038 23793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:24.560081 23799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:24.560855 23786 server_base.cc:1034] running on GCE node
I20260812 11:24:24.561617 23786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:24.563059 23786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:24.564321 23786 hybrid_clock.cc:648] HybridClock initialized: now 1786533864564257 us; error 39 us; skew 500 ppm
I20260812 11:24:24.564699 23786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:24 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533864, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:24.568087 23786 init.cc:377] Logged in from keytab as oryx/127.12.110.253@KRBTEST.COM (short username oryx)
I20260812 11:24:24.569677 23786 webserver.cc:460] Webserver started at http://127.12.110.253:41537/ using document root <none> and password file <none>
I20260812 11:24:24.570374 23786 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:24.570510 23786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:24.570897 23786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:24.572741 23576 heartbeater.cc:503] Master 127.12.110.254:44299 was elected leader, sending a full tablet report...
I20260812 11:24:24.574000 23786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/instance:
uuid: "d0e3d3fc75d5404a88526c3e5a7721a3"
format_stamp: "Formatted at 2026-08-12 11:24:24 on dist-test-slave-h6n0"
server_key: "597fcb62a1054cb17a76df333597064d"
server_key_iv: "d8c0f8721ce40b2e0e0ada3e7c9b6e9b"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.574923 23786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal/instance:
uuid: "d0e3d3fc75d5404a88526c3e5a7721a3"
format_stamp: "Formatted at 2026-08-12 11:24:24 on dist-test-slave-h6n0"
server_key: "597fcb62a1054cb17a76df333597064d"
server_key_iv: "d8c0f8721ce40b2e0e0ada3e7c9b6e9b"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.580036 23786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 11:24:24.583714 23811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:24.585078 23786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:24:24.585295 23786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
uuid: "d0e3d3fc75d5404a88526c3e5a7721a3"
format_stamp: "Formatted at 2026-08-12 11:24:24 on dist-test-slave-h6n0"
server_key: "597fcb62a1054cb17a76df333597064d"
server_key_iv: "d8c0f8721ce40b2e0e0ada3e7c9b6e9b"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.585590 23786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:24.606705 23786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:24.611452 23786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:24.611749 23786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:24.632882 23786 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.253:37647
I20260812 11:24:24.632903 23893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.253:37647 every 8 connection(s)
I20260812 11:24:24.634349 23786 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
I20260812 11:24:24.637869 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:24.640933 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 23786
I20260812 11:24:24.645753 23894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:24:24 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.253@KRBTEST.COM
Aug 12 11:24:24 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533864, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.253@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:24.664736 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.649237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.12.110.254:44299)
0812 11:24:24.649754 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.649761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.650185 (+ 424us) server_negotiation.cc:407] Connection header received
0812 11:24:24.650402 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.650408 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.650479 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.650628 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.651960 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.652799 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.654688 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.655092 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.655185 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.658902 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.658907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.658910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.658961 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.661817 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.662525 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.662554 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.662557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.662704 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.663097 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.663105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.663109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.663442 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.663602 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.664140 (+ 538us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.664369 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":261,"threads_started":1}
I20260812 11:24:24.665717 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.643400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40876 (local address 127.12.110.253:37647)
0812 11:24:24.644168 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.644179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.644216 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:24:24.645994 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.646015 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.646469 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.647026 (+ 557us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.648847 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.651064 (+ 2217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.652576 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.653375 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.653521 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.658505 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.658513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.658576 (+ 63us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.658620 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.661431 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.662134 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.662143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.662147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.662317 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.662791 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.662796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.662800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.663641 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.663994 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.664357 (+ 363us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.664716 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":575,"thread_start_us":355,"threads_started":1}
I20260812 11:24:24.665762 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.649338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:35134)
0812 11:24:24.650050 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:24.650083 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:24.650273 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:24.650689 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:24.650696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:24.651469 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:24.651794 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.651806 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.652946 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.652949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.654462 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.654477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.655341 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.655351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.655569 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.656186 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:24.656230 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:24.658710 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:24.661953 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.661967 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.661980 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.662406 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.662825 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.662830 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.662834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.662984 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.663575 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:24.663583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:24.663778 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 11:24:24.664049 (+ 271us) client_negotiation.cc:240] Negotiation successful
0812 11:24:24.664568 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"spinlock_wait_cycles":25984,"thread_start_us":379,"threads_started":1}
I20260812 11:24:24.667758 23894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:24:24 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533864, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.253@KRBTEST.COM for oryx/127.12.110.253@KRBTEST.COM
I20260812 11:24:24.680718 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.668630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.0.0.1:40886)
0812 11:24:24.668798 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:24.668820 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:24.668927 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:24.669218 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:24.669221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:24.669523 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:24.669650 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.669658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.670716 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.670719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.671559 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.671569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.672253 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.672258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.672407 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.673186 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:24.673204 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:24.675319 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:24.678134 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.678146 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.678153 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.678757 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.679131 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.679137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.679143 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.679338 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.679852 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:24.679859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:24.680026 (+ 167us) client_negotiation.cc:769] Sending connection context
0812 11:24:24.680193 (+ 167us) client_negotiation.cc:240] Negotiation successful
0812 11:24:24.680435 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:24:24.680891 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.668551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.12.110.253:37647)
0812 11:24:24.668733 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.668739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.668903 (+ 164us) server_negotiation.cc:407] Connection header received
0812 11:24:24.669032 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.669037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.669105 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.669180 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.669800 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.670513 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.671689 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.672091 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.672189 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.675508 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.675513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.675516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.675544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.677978 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.678884 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.678888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.678890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.679015 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.679448 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.679451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.679453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.679732 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.679843 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.680278 (+ 435us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.680564 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":108,"server-negotiator.queue_time_us":61}
I20260812 11:24:24.685882 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Bootstrap starting.
I20260812 11:24:24.689154 23894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:24.690239 23894 log.cc:824] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:24.693389 23894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: No bootstrap required, opened a new log
I20260812 11:24:24.700683 23894 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } }
I20260812 11:24:24.701052 23894 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:24.701172 23894 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e3d3fc75d5404a88526c3e5a7721a3, State: Initialized, Role: FOLLOWER
I20260812 11:24:24.701747 23894 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } }
I20260812 11:24:24.702728 23903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } } }
I20260812 11:24:24.703122 23903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:24.703893 23894 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: configured and running, proceeding with master startup.
W20260812 11:24:24.713156 23916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:24:24.713307 23916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:24:24.716498 23786 master_runner.cc:428] Detected that this master d0e3d3fc75d5404a88526c3e5a7721a3 is joining an existing cluster
I20260812 11:24:24.716617 23786 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.110.253:37647
I20260812 11:24:24.739646 23923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.720623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:35144)
0812 11:24:24.721863 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:24.721885 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:24.722078 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:24.722699 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:24.722705 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:24.723515 (+ 810us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:24.723990 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.724007 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.725585 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.725592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.728416 (+ 2824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.728436 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.729807 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.729832 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.730056 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.730882 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:24.730910 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:24.732163 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:24.736138 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.736146 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.736157 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.736696 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.737158 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.737165 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.737172 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.737382 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.738130 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:24.738137 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:24.738388 (+ 251us) client_negotiation.cc:769] Sending connection context
0812 11:24:24.738591 (+ 203us) client_negotiation.cc:240] Negotiation successful
0812 11:24:24.738967 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"thread_start_us":873,"threads_started":1}
I20260812 11:24:24.740675 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.721115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35144 (local address 127.12.110.254:44299)
0812 11:24:24.721342 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.721348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.721968 (+ 620us) server_negotiation.cc:407] Connection header received
0812 11:24:24.722242 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.722251 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.722349 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.722429 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.724202 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.725350 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.728590 (+ 3240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.729358 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.729473 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.732371 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.732379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.732387 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.732431 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.736002 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.736845 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.736853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.736859 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.737024 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.737516 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.737524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.737531 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.737920 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.738067 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.739980 (+ 1913us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.740228 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 11:24:24.747773 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.720813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.12.110.253:37647)
0812 11:24:24.720970 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.720979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.731978 (+ 10999us) server_negotiation.cc:407] Connection header received
0812 11:24:24.732372 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.732378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.732439 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.732496 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.733978 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.735085 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.736983 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.737698 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.737790 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.740235 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.740241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.740245 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.740278 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.744159 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.745049 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.745054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.745059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.745220 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.745646 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.745651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.745655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.746210 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.746328 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.746985 (+ 657us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.747270 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 11:24:24.749106 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.720629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.0.0.1:40902)
0812 11:24:24.731869 (+ 11240us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:24.731894 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:24.732246 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:24.732668 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:24.732673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:24.733462 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:24.733817 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.733830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.735308 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.735313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.736834 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.736849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.738108 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.738117 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.738292 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.738976 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:24.738997 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:24.740060 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:24.744354 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.744359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.744363 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.744916 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.745360 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.745367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.745372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.745535 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.746417 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:24.746421 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:24.746627 (+ 206us) client_negotiation.cc:769] Sending connection context
0812 11:24:24.748281 (+ 1654us) client_negotiation.cc:240] Negotiation successful
0812 11:24:24.748585 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11106,"mutex_wait_us":52,"thread_start_us":11295,"threads_started":1}
WARNING: no policy specified for oryx/127.12.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.252@KRBTEST.COM" created.
Aug 12 11:24:24 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533862, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.254@KRBTEST.COM for oryx/127.12.110.253@KRBTEST.COM
I20260812 11:24:24.767231 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.748457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.12.110.253:37647)
0812 11:24:24.750243 (+ 1786us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.750251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.751654 (+ 1403us) server_negotiation.cc:407] Connection header received
0812 11:24:24.752059 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.752066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.752191 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.752314 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.753812 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.754984 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.756983 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.757626 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.757805 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.761459 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.761464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.761467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.761499 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.764344 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.765095 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.765100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.765104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.765244 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.765637 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.765642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.765646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.765923 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.766076 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.766601 (+ 525us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.766829 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1645}
I20260812 11:24:24.767278 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.748133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.0.0.1:40906)
0812 11:24:24.751520 (+ 3387us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:24.751566 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:24.751799 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:24.752397 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:24.752405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:24.753271 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:24.753581 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.753595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.755167 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.755172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.756804 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.756821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.757830 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.757837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.758060 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.758782 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:24.758806 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:24.761257 (+ 2451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:24.764528 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.764540 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.764548 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.764967 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.765362 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.765370 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.765375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.765525 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.766069 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:24.766078 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:24.766253 (+ 175us) client_negotiation.cc:769] Sending connection context
0812 11:24:24.766460 (+ 207us) client_negotiation.cc:240] Negotiation successful
0812 11:24:24.766831 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3120,"thread_start_us":2977,"threads_started":1}
I20260812 11:24:24.769598 23037 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.110.253:37647: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3"
member_type: NON_VOTER
last_known_addr {
host: "127.12.110.253"
port: 37647
}
attrs {
promote: true
}
}
dest_uuid: "5cde5ff1cbff4d0f8274ef992177d97e"
cas_config_opid_index: -1
I20260812 11:24:24.770674 23037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } }
I20260812 11:24:24.787676 23856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:24.787318 23935 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d0e3d3fc75d5404a88526c3e5a7721a3 (127.12.110.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:24.791026 23856 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Refusing update from remote peer 5cde5ff1cbff4d0f8274ef992177d97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260812 11:24:24.792021 23931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:24.796859 23935 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.110.253:37647
I20260812 11:24:24.798940 23786 master.cc:537] Master@127.12.110.253:37647 shutting down...
I20260812 11:24:24.800547 23936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:24.801018 23936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.252.keytab.
Entry for principal oryx/127.12.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.252.keytab.
I20260812 11:24:24.804821 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:34385
--webserver_interface=127.12.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.252.keytab
--principal=oryx/127.12.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:24:24.825470 23856 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d0e3d3fc75d5404a88526c3e5a7721a3 (127.12.110.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:24.830149 23950 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:24.830650 23950 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:24:24.831256 23903 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde5ff1cbff4d0f8274ef992177d97e. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:24.831537 23903 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:24:24.831857 23944 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:24.832070 23944 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:24:24.857048 23786 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 LEARNER]: Raft consensus shutting down.
I20260812 11:24:24.857695 23786 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 LEARNER]: Raft consensus is shut down!
I20260812 11:24:24.857896 23786 tablet_replica.cc:330] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: stopping tablet replica
I20260812 11:24:24.867219 23786 master.cc:552] Master@127.12.110.253:37647 shutdown complete.
I20260812 11:24:24.867405 23786 master_runner.cc:305] Clearing existing system tablet
I20260812 11:24:24.867893 23786 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:24.868023 23786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:24.870638 23786 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260812 11:24:24.872367 23953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:24.872802 23786 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:24.873178 23786 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260812 11:24:24.873281 23786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
uuid: "d0e3d3fc75d5404a88526c3e5a7721a3"
format_stamp: "Formatted at 2026-08-12 11:24:24 on dist-test-slave-h6n0"
server_key: "597fcb62a1054cb17a76df333597064d"
server_key_iv: "d8c0f8721ce40b2e0e0ada3e7c9b6e9b"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.877697 23786 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:24.884249 23786 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 11:24:24.884531 23786 log.cc:1196] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260812 11:24:24.885515 23786 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Deleting consensus metadata
I20260812 11:24:24.885931 23786 master_runner.cc:315] Copying system tablet from 127.12.110.254:44299
I20260812 11:24:24.889726 23786 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: tablet copy: Beginning tablet copy session from remote peer at address 127.12.110.254:44299
I20260812 11:24:24.903846 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.890613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35148 (local address 127.12.110.254:44299)
0812 11:24:24.890836 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:24.890845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:24.891433 (+ 588us) server_negotiation.cc:407] Connection header received
0812 11:24:24.891668 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:24.891673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:24.891755 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:24.891862 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:24.892734 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.893465 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.894940 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.895398 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.895572 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.897783 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:24.897789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:24.897794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:24.897833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:24.901052 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.901740 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.901746 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.901749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.901882 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:24.902273 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:24.902282 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:24.902284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:24.902625 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:24.902777 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:24.903226 (+ 449us) server_negotiation.cc:299] Negotiation successful
0812 11:24:24.903458 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:24:24.903985 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:24.890603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:35148)
0812 11:24:24.891320 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:24.891336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:24.891505 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:24.891912 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:24.891917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:24.892346 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:24.892575 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.892588 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.893612 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.893618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.894773 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:24.894789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:24.895788 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:24.895793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:24.895988 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:24.896670 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:24.896692 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:24.897590 (+ 898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:24.901208 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.901216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.901220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.901594 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.901998 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:24.902003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:24.902007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:24.902149 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:24.902742 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:24.902747 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:24.902937 (+ 190us) client_negotiation.cc:769] Sending connection context
0812 11:24:24.903108 (+ 171us) client_negotiation.cc:240] Negotiation successful
0812 11:24:24.903428 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":423,"threads_started":1}
I20260812 11:24:24.905076 23069 tablet_copy_service.cc:140] P 5cde5ff1cbff4d0f8274ef992177d97e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d0e3d3fc75d5404a88526c3e5a7721a3 ({username='oryx', principal='oryx/127.12.110.253@KRBTEST.COM'} at 127.0.0.1:35148)
I20260812 11:24:24.905350 23069 tablet_copy_service.cc:161] P 5cde5ff1cbff4d0f8274ef992177d97e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d0e3d3fc75d5404a88526c3e5a7721a3 at {username='oryx', principal='oryx/127.12.110.253@KRBTEST.COM'} at 127.0.0.1:35148: session id = d0e3d3fc75d5404a88526c3e5a7721a3-00000000000000000000000000000000
I20260812 11:24:24.908900 23069 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 11:24:24.911038 23786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 11:24:24.916273 23786 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: tablet copy: Starting download of 0 data blocks...
I20260812 11:24:24.916636 23786 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: tablet copy: Starting download of 1 WAL segments...
I20260812 11:24:24.919111 23786 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 11:24:24.923393 23069 tablet_copy_service.cc:342] P 5cde5ff1cbff4d0f8274ef992177d97e: Request end of tablet copy session d0e3d3fc75d5404a88526c3e5a7721a3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.110.253@KRBTEST.COM'} at 127.0.0.1:35148
I20260812 11:24:24.923586 23069 tablet_copy_service.cc:433] P 5cde5ff1cbff4d0f8274ef992177d97e: ending tablet copy session d0e3d3fc75d5404a88526c3e5a7721a3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d0e3d3fc75d5404a88526c3e5a7721a3
I20260812 11:24:24.933151 23786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:24.935717 23962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:24.940093 23964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:24.947976 23786 server_base.cc:1034] running on GCE node
W20260812 11:24:24.948140 23966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:24.948675 23786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:24.949407 23786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:24.950835 23786 hybrid_clock.cc:648] HybridClock initialized: now 1786533864950740 us; error 46 us; skew 500 ppm
I20260812 11:24:24.951083 23786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:24 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533864, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:24.953074 23786 init.cc:377] Logged in from keytab as oryx/127.12.110.253@KRBTEST.COM (short username oryx)
I20260812 11:24:24.954777 23786 webserver.cc:460] Webserver started at http://127.12.110.253:32933/ using document root <none> and password file <none>
I20260812 11:24:24.955143 23786 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:24.955317 23786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:24.957957 23786 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260812 11:24:24.959764 23977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:24.960297 23786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 11:24:24.960431 23786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
uuid: "d0e3d3fc75d5404a88526c3e5a7721a3"
format_stamp: "Formatted at 2026-08-12 11:24:24 on dist-test-slave-h6n0"
server_key: "597fcb62a1054cb17a76df333597064d"
server_key_iv: "d8c0f8721ce40b2e0e0ada3e7c9b6e9b"
server_key_version: "encryptionkey@0"
I20260812 11:24:24.960613 23786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:24.992223 23786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:24.996950 23786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:25.038180 23786 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.253:37647
I20260812 11:24:25.038203 24058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.253:37647 every 8 connection(s)
I20260812 11:24:25.039193 23786 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
I20260812 11:24:25.046348 24059 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:24:25.054303 24059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Bootstrap starting.
I20260812 11:24:25.065827 24059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:24:25.066277 24059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: Bootstrap complete.
I20260812 11:24:25.067209 24059 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } }
I20260812 11:24:25.067440 24059 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d0e3d3fc75d5404a88526c3e5a7721a3, State: Initialized, Role: LEARNER
I20260812 11:24:25.067713 24059 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } }
I20260812 11:24:25.068874 24059 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:25.068645 24063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:25.069108 24063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:24:25.081576 24082 catalog_manager.cc:1259] Loaded cluster ID: 490a3b38963f4c17bd8d048ac6a3c817
I20260812 11:24:25.081697 24082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: loading cluster ID for follower catalog manager: success
I20260812 11:24:25.087622 24082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: acquiring CA information for follower catalog manager: success
I20260812 11:24:25.090929 24082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:24:25.121045 23780 heartbeater.cc:503] Master 127.12.110.254:44299 was elected leader, sending a full tablet report...
W20260812 11:24:25.250705 23942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:25.250978 23942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:25.251127 23942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260812 11:24:25.251811 23931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:25.252164 23931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:25.260872 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.252205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.0.0.1:40908)
0812 11:24:25.252490 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.252514 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.252644 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.253848 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.253853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.253893 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:25.254101 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.254113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.256328 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.256334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.258383 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.258398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.260100 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.260106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.260318 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.260373 (+ 55us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.260464 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.260597 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260812 11:24:25.261169 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.252704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.12.110.253:37647)
0812 11:24:25.253412 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.253418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.253444 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:24:25.253519 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.253526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.253640 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.253769 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:25.254283 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.256155 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.258584 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.259908 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.260096 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.260274 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.260518 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.260648 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":357,"threads_started":1}
W20260812 11:24:25.261399 23942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:25.261598 23942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:25.261736 23942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:25.261845 23942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:25.261889 23942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:25.262472 24063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde5ff1cbff4d0f8274ef992177d97e. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:25.262771 24063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: LEARNER
I20260812 11:24:25.263622 23935 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: attempting to promote NON_VOTER d0e3d3fc75d5404a88526c3e5a7721a3 to VOTER
I20260812 11:24:25.264583 23935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } }
I20260812 11:24:25.266425 24024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 LEARNER]: Refusing update from remote peer 5cde5ff1cbff4d0f8274ef992177d97e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260812 11:24:25.267081 23931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260812 11:24:25.270162 23935 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0e3d3fc75d5404a88526c3e5a7721a3 (127.12.110.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } }
I20260812 11:24:25.271270 24024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d0e3d3fc75d5404a88526c3e5a7721a3 (127.12.110.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } }
I20260812 11:24:25.276482 24063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } }
I20260812 11:24:25.276752 24063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:25.277364 23942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.252.keytab
--principal=oryx/127.12.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:34385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:25.277521 23936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } }
I20260812 11:24:25.277760 23936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:25.277822 23931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } }
I20260812 11:24:25.278031 23931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:25.279029 23942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:25.280333 23942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:25.292763 24091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:25.292683 24094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:25.292829 24096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:25.293869 23942 server_base.cc:1034] running on GCE node
I20260812 11:24:25.294832 23942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:25.296504 23942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:25.297740 23942 hybrid_clock.cc:648] HybridClock initialized: now 1786533865297675 us; error 58 us; skew 500 ppm
I20260812 11:24:25.298166 23942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533865, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:25.301926 23942 init.cc:377] Logged in from keytab as oryx/127.12.110.252@KRBTEST.COM (short username oryx)
I20260812 11:24:25.303632 23942 webserver.cc:460] Webserver started at http://127.12.110.252:40141/ using document root <none> and password file <none>
I20260812 11:24:25.304379 23942 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:25.304557 23942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:25.304895 23942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:25.308185 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/instance:
uuid: "6dc45d4b35ca438a8a852f366e9aef2e"
format_stamp: "Formatted at 2026-08-12 11:24:25 on dist-test-slave-h6n0"
server_key: "fddd27058a37188c1f6029763970fa22"
server_key_iv: "bfbd9bbc27db6e492d71668ef210c1a1"
server_key_version: "encryptionkey@0"
I20260812 11:24:25.309043 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal/instance:
uuid: "6dc45d4b35ca438a8a852f366e9aef2e"
format_stamp: "Formatted at 2026-08-12 11:24:25 on dist-test-slave-h6n0"
server_key: "fddd27058a37188c1f6029763970fa22"
server_key_iv: "bfbd9bbc27db6e492d71668ef210c1a1"
server_key_version: "encryptionkey@0"
I20260812 11:24:25.314656 23942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 11:24:25.318243 24105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:25.319868 23942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:24:25.320043 23942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
uuid: "6dc45d4b35ca438a8a852f366e9aef2e"
format_stamp: "Formatted at 2026-08-12 11:24:25 on dist-test-slave-h6n0"
server_key: "fddd27058a37188c1f6029763970fa22"
server_key_iv: "bfbd9bbc27db6e492d71668ef210c1a1"
server_key_version: "encryptionkey@0"
I20260812 11:24:25.320261 23942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:25.332257 23942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:25.336515 23942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:25.336809 23942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:25.355980 23942 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.252:34385
I20260812 11:24:25.356020 24190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.252:34385 every 8 connection(s)
I20260812 11:24:25.357313 23942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
I20260812 11:24:25.361127 24191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:25.364818 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 23942
I20260812 11:24:25.368373 24191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.252@KRBTEST.COM
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533865, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.252@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:25.389564 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.372490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35164 (local address 127.12.110.254:44299)
0812 11:24:25.372659 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.372663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.373337 (+ 674us) server_negotiation.cc:407] Connection header received
0812 11:24:25.373535 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.373541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.373617 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.373723 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.375155 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.376097 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.378223 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.378786 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.378904 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.382771 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.382795 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.382805 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.382850 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.386716 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.387427 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.387438 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.387442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.387573 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.388018 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.388025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.388028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.388324 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.388483 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.389016 (+ 533us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.389230 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:24:25.390202 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.366958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.12.110.252:34385)
0812 11:24:25.367762 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.367779 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.367828 (+ 49us) server_negotiation.cc:407] Connection header received
0812 11:24:25.369647 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.369676 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.370187 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.370639 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.372000 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.373911 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.375678 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.376596 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.376729 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.382162 (+ 5433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.382171 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.382187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.382224 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.385687 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.386587 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.386594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.386600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.386752 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.387178 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.387182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.387184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.387774 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.388134 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.388612 (+ 478us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.389061 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":345,"threads_started":1}
I20260812 11:24:25.390295 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.372473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:35164)
0812 11:24:25.373182 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.373228 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.373407 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.373780 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.373786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.374653 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.374991 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.375004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.376286 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.376292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.378055 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.378075 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.379049 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.379059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.379304 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.379972 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.380002 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.382513 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.386867 (+ 4354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.386878 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.386885 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.387294 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.387715 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.387720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.387724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.387902 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.388466 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.388473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.388643 (+ 170us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.388897 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.389170 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":376,"threads_started":1}
I20260812 11:24:25.392640 24191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533865, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.252@KRBTEST.COM for oryx/127.12.110.253@KRBTEST.COM
I20260812 11:24:25.406719 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.393619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.0.0.1:40924)
0812 11:24:25.393821 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.393838 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.393992 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.394653 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.394657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.395008 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.395158 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.395166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.396225 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.396229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.397164 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.397175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.397878 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.397883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.398025 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.398716 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.398735 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.401175 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.404286 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.404292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.404296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.404692 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.405075 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.405080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.405083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.405228 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.405738 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.405742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.405890 (+ 148us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.406034 (+ 144us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.406345 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 11:24:25.406730 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.394110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40924 (local address 127.12.110.253:37647)
0812 11:24:25.394301 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.394305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.394323 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:24:25.394375 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.394390 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.394449 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.394592 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.395325 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.396068 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.397312 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.397730 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.397863 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.401376 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.401381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.401384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.401413 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.404148 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.404813 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.404835 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.404838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.404960 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.405340 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.405348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.405352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.405616 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.405769 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.406126 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.406353 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:24:25.407904 24191 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533865, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.252@KRBTEST.COM for oryx/127.12.110.252@KRBTEST.COM
I20260812 11:24:25.420799 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.408649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.0.0.1:34686)
0812 11:24:25.408880 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.408900 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.409001 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.409251 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.409254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.409591 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.409734 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.409742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.410749 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.410753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.411575 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.411585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.412216 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.412220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.412366 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.412876 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.412891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.414886 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.417958 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.417962 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.417964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.418334 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.419111 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.419115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.419117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.419283 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.419951 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.419956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.420103 (+ 147us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.420241 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.420452 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 11:24:25.421176 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.408864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.12.110.252:34385)
0812 11:24:25.409012 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.409016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.409054 (+ 38us) server_negotiation.cc:407] Connection header received
0812 11:24:25.409090 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.409093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.409144 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.409212 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.409881 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.410638 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.411690 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.412088 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.412179 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.415045 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.415049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.415052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.415079 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.417837 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.418667 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.418674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.418677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.418933 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.419432 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.419436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.419439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.419785 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.419911 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.420487 (+ 576us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.420736 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:24:25.426926 24191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Bootstrap starting.
I20260812 11:24:25.430785 24191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:25.431850 24191 log.cc:824] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:25.434417 24191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: No bootstrap required, opened a new log
I20260812 11:24:25.441514 24191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } }
I20260812 11:24:25.441906 24191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:25.442000 24191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dc45d4b35ca438a8a852f366e9aef2e, State: Initialized, Role: FOLLOWER
I20260812 11:24:25.442627 24191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } }
I20260812 11:24:25.443539 24202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } } }
I20260812 11:24:25.443912 24202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:25.444840 24191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: configured and running, proceeding with master startup.
W20260812 11:24:25.455574 24223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:24:25.455725 24223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:24:25.459764 23942 master_runner.cc:428] Detected that this master 6dc45d4b35ca438a8a852f366e9aef2e is joining an existing cluster
I20260812 11:24:25.459935 23942 master_runner.cc:432] Initiating AddMaster RPC to add 127.12.110.252:34385
I20260812 11:24:25.491544 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.464244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.12.110.252:34385)
0812 11:24:25.464473 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.464482 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.474505 (+ 10023us) server_negotiation.cc:407] Connection header received
0812 11:24:25.474690 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.474695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.474771 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.474848 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.476390 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.477661 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.479886 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.480694 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.480790 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.483947 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.483952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.483956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.484007 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.488269 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.489167 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.489171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.489174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.489330 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.489781 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.489785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.489788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.490157 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.490307 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.490905 (+ 598us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.491148 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":62}
I20260812 11:24:25.491539 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.465247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.12.110.254:44299)
0812 11:24:25.465487 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.465496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.473449 (+ 7953us) server_negotiation.cc:407] Connection header received
0812 11:24:25.473971 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.473978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.474080 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.474164 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.476012 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.477267 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.479893 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.480591 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.480688 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.483983 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.483990 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.483994 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.484031 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.488385 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.489177 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.489183 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.489186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.489338 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.489797 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.489801 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.489804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.490141 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.490292 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.490944 (+ 652us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.491149 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:24:25.492703 24233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.464819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.0.0.1:34702)
0812 11:24:25.474391 (+ 9572us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.474406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.474557 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.475064 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.475068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.475871 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.476254 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.476268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.477886 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.477893 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.479734 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.479748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.481048 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.481053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.481199 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.482200 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.482219 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.483739 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.488405 (+ 4666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.488411 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.488416 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.489042 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.489474 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.489479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.489482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.489657 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.490370 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.490375 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.490629 (+ 254us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.491815 (+ 1186us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.492153 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9440,"mutex_wait_us":90,"spinlock_wait_cycles":384,"thread_start_us":8202,"threads_started":1}
I20260812 11:24:25.493088 24088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.464845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.12.110.253:37647)
0812 11:24:25.465037 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.465043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.473889 (+ 8846us) server_negotiation.cc:407] Connection header received
0812 11:24:25.474847 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.474853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.474942 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.475023 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.477375 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.478446 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.480713 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.481389 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.481492 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.485403 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.485409 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.485413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.485451 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.489749 (+ 4298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.490603 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.490608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.490612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.490771 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.491256 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.491261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.491265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.491620 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.491748 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.492470 (+ 722us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.492708 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:24:25.493561 24234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.464221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:35174)
0812 11:24:25.473324 (+ 9103us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.473345 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.473793 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.474390 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.474394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.475257 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.475816 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.475836 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.477489 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.477494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.479734 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.479748 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.480978 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.480984 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.481197 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.482143 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.482175 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.483788 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.488532 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.488539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.488545 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.489042 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.489510 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.489516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.489520 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.489671 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.490371 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.490375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.490630 (+ 255us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.492294 (+ 1664us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.492589 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8959,"mutex_wait_us":90,"spinlock_wait_cycles":63488,"thread_start_us":9257,"threads_started":1}
I20260812 11:24:25.494163 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.464360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.0.0.1:40930)
0812 11:24:25.473799 (+ 9439us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.473814 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.474651 (+ 837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.475295 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.475302 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.476172 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.477022 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.477058 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.478685 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.478691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.480536 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.480552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.482369 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.482379 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.482586 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.483268 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.483290 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.485189 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.489928 (+ 4739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.489933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.489937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.490427 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.490934 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.490939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.490943 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.491120 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.491866 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.491874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.492151 (+ 277us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.493331 (+ 1180us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.493653 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9299,"mutex_wait_us":58,"thread_start_us":9691,"threads_started":1}
I20260812 11:24:25.497735 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 23136
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533862, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.254@KRBTEST.COM for oryx/127.12.110.252@KRBTEST.COM
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.193@KRBTEST.COM
I20260812 11:24:25.514904 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.500836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.12.110.252:34385)
0812 11:24:25.501238 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.501242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.501254 (+ 12us) server_negotiation.cc:407] Connection header received
0812 11:24:25.501315 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.501318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.501392 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.501501 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.502227 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.503005 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.504636 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.505361 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.505500 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.508695 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.508700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.508703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.508731 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.512232 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.513004 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.513011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.513019 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.513153 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.513535 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.513541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.513544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.513794 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.513944 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.514323 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.514522 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260812 11:24:25.514918 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.500722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.0.0.1:34712)
0812 11:24:25.501003 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.501017 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.501094 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.501543 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.501547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.501921 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.502072 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.502080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.503134 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.503139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.504479 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.504493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.505533 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.505541 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.505696 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.506359 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.506387 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.508510 (+ 2123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.512377 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.512382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.512386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.512865 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.513273 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.513279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.513283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.513417 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.513914 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.513919 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.514079 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.514222 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.514508 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260812 11:24:25.516211 23037 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.12.110.252:34385: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e"
member_type: NON_VOTER
last_known_addr {
host: "127.12.110.252"
port: 34385
}
attrs {
promote: true
}
}
dest_uuid: "5cde5ff1cbff4d0f8274ef992177d97e"
cas_config_opid_index: 6
I20260812 11:24:25.517036 23037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } }
I20260812 11:24:25.519348 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.498814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.12.110.193:46721)
0812 11:24:25.499484 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.499492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.499521 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:24:25.499597 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.499607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.499887 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.500056 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.501733 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.503343 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.504985 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.505657 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.505780 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.511557 (+ 5777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.511571 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.511590 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.511633 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.516095 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.516907 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.516916 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.516920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.517083 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.517506 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.517511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.517515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.517962 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.518284 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.518620 (+ 336us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.518911 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":1684,"threads_started":1}
I20260812 11:24:25.520082 24024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Refusing update from remote peer 5cde5ff1cbff4d0f8274ef992177d97e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 11:24:25.521144 23346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:25.521462 24151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:25.526811 23931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260812 11:24:25.528131 24151 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 FOLLOWER]: Refusing update from remote peer 5cde5ff1cbff4d0f8274ef992177d97e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260812 11:24:25.529143 23931 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:25.530035 23935 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dc45d4b35ca438a8a852f366e9aef2e (127.12.110.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:25.533906 23935 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.12.110.252:34385
I20260812 11:24:25.534617 23931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:25.534996 23931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:25.539384 23942 master.cc:537] Master@127.12.110.252:34385 shutting down...
I20260812 11:24:25.540758 24024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dc45d4b35ca438a8a852f366e9aef2e (127.12.110.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:25.545929 23931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:25.546180 23931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:25.546723 24063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:25.546960 24063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:25.554875 24150 raft_consensus.cc:858] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } }
W20260812 11:24:25.555081 24150 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } }
W20260812 11:24:25.589047 24114 tablet.cc:2369] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:25.606946 24150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d0e3d3fc75d5404a88526c3e5a7721a3 (127.12.110.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:25.611966 24202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde5ff1cbff4d0f8274ef992177d97e. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:25.612238 24202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 11:24:25.612440 24202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:25.612646 24202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 11:24:25.612977 24239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: NON_VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: true } } }
I20260812 11:24:25.613281 24239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260812 11:24:25.613116 23935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:25.613461 23935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
W20260812 11:24:25.615381 23017 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260812 11:24:25.617406 23942 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260812 11:24:25.617529 23017 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e -> Peer 6dc45d4b35ca438a8a852f366e9aef2e (127.12.110.252:34385): Couldn't send request to peer 6dc45d4b35ca438a8a852f366e9aef2e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260812 11:24:25.617908 23942 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260812 11:24:25.618119 23942 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: stopping tablet replica
I20260812 11:24:25.627210 23942 master.cc:552] Master@127.12.110.252:34385 shutdown complete.
I20260812 11:24:25.627331 23942 master_runner.cc:305] Clearing existing system tablet
I20260812 11:24:25.627811 23942 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:25.627967 23942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:25.631320 23942 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260812 11:24:25.632864 24245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:25.633198 23942 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:25.633490 23942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260812 11:24:25.633610 23942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
uuid: "6dc45d4b35ca438a8a852f366e9aef2e"
format_stamp: "Formatted at 2026-08-12 11:24:25 on dist-test-slave-h6n0"
server_key: "fddd27058a37188c1f6029763970fa22"
server_key_iv: "bfbd9bbc27db6e492d71668ef210c1a1"
server_key_version: "encryptionkey@0"
I20260812 11:24:25.636837 23942 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:25.641708 23942 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260812 11:24:25.641899 23942 log.cc:1196] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260812 11:24:25.642439 23942 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Deleting consensus metadata
I20260812 11:24:25.642746 23942 master_runner.cc:315] Copying system tablet from 127.12.110.254:44299
I20260812 11:24:25.645679 23942 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: tablet copy: Beginning tablet copy session from remote peer at address 127.12.110.254:44299
I20260812 11:24:25.659935 23899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.646668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.12.110.254:44299)
0812 11:24:25.646972 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:25.646981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:25.647582 (+ 601us) server_negotiation.cc:407] Connection header received
0812 11:24:25.647859 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:25.647864 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:25.647946 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:25.648076 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:25.648998 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.649733 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.651393 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.651838 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.651979 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.653993 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:25.654003 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:25.654007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:25.654038 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:25.657122 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.657810 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.657817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.657820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.657961 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:25.658362 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:25.658368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:25.658372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:25.658734 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:25.658894 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:25.659299 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 11:24:25.659527 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260812 11:24:25.659996 24260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:25.646514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:35188)
0812 11:24:25.647466 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:25.647482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:25.647651 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:25.648112 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:25.648117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:25.648579 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:25.648808 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.648819 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.649886 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.649891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.651222 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:25.651236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:25.651994 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:25.652000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:25.652196 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:25.652905 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:25.652933 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:25.653787 (+ 854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:25.657270 (+ 3483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.657275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.657279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.657681 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.658083 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:25.658089 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:25.658092 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:25.658241 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:25.658855 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:25.658860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:25.659055 (+ 195us) client_negotiation.cc:769] Sending connection context
0812 11:24:25.659203 (+ 148us) client_negotiation.cc:240] Negotiation successful
0812 11:24:25.659576 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":587,"threads_started":1}
I20260812 11:24:25.660933 23069 tablet_copy_service.cc:140] P 5cde5ff1cbff4d0f8274ef992177d97e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6dc45d4b35ca438a8a852f366e9aef2e ({username='oryx', principal='oryx/127.12.110.252@KRBTEST.COM'} at 127.0.0.1:35188)
I20260812 11:24:25.661137 23069 tablet_copy_service.cc:161] P 5cde5ff1cbff4d0f8274ef992177d97e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6dc45d4b35ca438a8a852f366e9aef2e at {username='oryx', principal='oryx/127.12.110.252@KRBTEST.COM'} at 127.0.0.1:35188: session id = 6dc45d4b35ca438a8a852f366e9aef2e-00000000000000000000000000000000
I20260812 11:24:25.664819 23069 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: Tablet Copy: opened 0 blocks and 1 log segments
I20260812 11:24:25.666682 23942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260812 11:24:25.673506 23942 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: tablet copy: Starting download of 0 data blocks...
I20260812 11:24:25.673832 23942 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: tablet copy: Starting download of 1 WAL segments...
I20260812 11:24:25.676486 23942 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260812 11:24:25.680897 23069 tablet_copy_service.cc:342] P 5cde5ff1cbff4d0f8274ef992177d97e: Request end of tablet copy session 6dc45d4b35ca438a8a852f366e9aef2e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.12.110.252@KRBTEST.COM'} at 127.0.0.1:35188
I20260812 11:24:25.681033 23069 tablet_copy_service.cc:433] P 5cde5ff1cbff4d0f8274ef992177d97e: ending tablet copy session 6dc45d4b35ca438a8a852f366e9aef2e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6dc45d4b35ca438a8a852f366e9aef2e
I20260812 11:24:25.694118 23942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260812 11:24:25.694840 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 23136
W20260812 11:24:25.704530 24261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:25.716197 24264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:25.724267 23942 server_base.cc:1034] running on GCE node
W20260812 11:24:25.724792 24267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:25.725311 23942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:25.726051 23942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:25.727270 23942 hybrid_clock.cc:648] HybridClock initialized: now 1786533865727063 us; error 184 us; skew 500 ppm
I20260812 11:24:25.727558 23942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:24:25.727741 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:46721
--local_ip_for_outbound_sockets=127.12.110.193
--tserver_master_addrs=127.12.110.254:44299
--webserver_port=37183
--webserver_interface=127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab
--principal=oryx/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:24:25 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533865, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:25.729645 23942 init.cc:377] Logged in from keytab as oryx/127.12.110.252@KRBTEST.COM (short username oryx)
I20260812 11:24:25.734930 23942 webserver.cc:460] Webserver started at http://127.12.110.252:45733/ using document root <none> and password file <none>
I20260812 11:24:25.735271 23942 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:25.735400 23942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:25.738102 23942 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260812 11:24:25.739820 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:25.740293 23942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260812 11:24:25.740422 23942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
uuid: "6dc45d4b35ca438a8a852f366e9aef2e"
format_stamp: "Formatted at 2026-08-12 11:24:25 on dist-test-slave-h6n0"
server_key: "fddd27058a37188c1f6029763970fa22"
server_key_iv: "bfbd9bbc27db6e492d71668ef210c1a1"
server_key_version: "encryptionkey@0"
I20260812 11:24:25.740617 23942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:25.790043 23942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:25.794013 23942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:25.825712 23942 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.252:34385
I20260812 11:24:25.825776 24362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.252:34385 every 8 connection(s)
I20260812 11:24:25.826884 23942 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
I20260812 11:24:25.832873 24364 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:24:25.836725 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Bootstrap starting.
I20260812 11:24:25.855872 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:24:25.856455 24364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Bootstrap complete.
I20260812 11:24:25.857628 24364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } }
I20260812 11:24:25.858060 24364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dc45d4b35ca438a8a852f366e9aef2e, State: Initialized, Role: LEARNER
I20260812 11:24:25.858355 24364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } }
I20260812 11:24:25.862996 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:25.863354 24364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:25.863502 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: LEARNER
I20260812 11:24:25.886473 24385 catalog_manager.cc:1259] Loaded cluster ID: 490a3b38963f4c17bd8d048ac6a3c817
I20260812 11:24:25.886620 24385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: loading cluster ID for follower catalog manager: success
I20260812 11:24:25.890023 24385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: acquiring CA information for follower catalog manager: success
I20260812 11:24:25.892386 24385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 11:24:26.136196 24279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:26.136482 24279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:26.136579 24279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:26.143787 24279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:26.143904 24279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:26.144017 24279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:26.154970 24279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab
--principal=oryx/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:46721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=37183
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:26.156489 24279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:26.158020 24279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:26.172016 24399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:26.172118 24398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:26.172292 24401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:26.173072 24279 server_base.cc:1034] running on GCE node
I20260812 11:24:26.173775 24279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:26.174760 24279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:26.176059 24279 hybrid_clock.cc:648] HybridClock initialized: now 1786533866176019 us; error 50 us; skew 500 ppm
I20260812 11:24:26.176440 24279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:26.179746 24279 init.cc:377] Logged in from keytab as oryx/127.12.110.193@KRBTEST.COM (short username oryx)
I20260812 11:24:26.181206 24279 webserver.cc:460] Webserver started at http://127.12.110.193:37183/ using document root <none> and password file <none>
I20260812 11:24:26.181933 24279 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:26.182101 24279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:26.187541 24279 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:24:26.189069 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:26.190138 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:26.192226 24422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:26.193712 24279 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260812 11:24:26.193902 24279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "bc8713ea524e4655b599a1619c945cc4"
format_stamp: "Formatted at 2026-08-12 11:24:22 on dist-test-slave-h6n0"
server_key: "daff1e624944caf5f309481522fe3c4f"
server_key_iv: "3572f28f94b12e9f5d260da261a39379"
server_key_version: "encryptionkey@0"
I20260812 11:24:26.195263 24279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:26.197701 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.189490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34716 (local address 127.12.110.252:34385)
0812 11:24:26.190140 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.190148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.191723 (+ 1575us) server_negotiation.cc:407] Connection header received
0812 11:24:26.191950 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.191954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.192027 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.192157 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:26.192616 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.193610 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.195946 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.196756 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.196909 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.197030 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.197334 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.197433 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":874,"threads_started":1}
I20260812 11:24:26.197762 24415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.189240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.0.0.1:34716)
0812 11:24:26.191583 (+ 2343us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:26.191611 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:26.191789 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:26.192175 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:26.192180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:26.192209 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:26.192448 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.192458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.193787 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.193793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.195759 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.195777 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.196953 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.196959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.197153 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.197188 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:24:26.197291 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 11:24:26.197430 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2167,"thread_start_us":1976,"threads_started":1}
I20260812 11:24:26.198629 24328 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260812 11:24:26.199065 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cde5ff1cbff4d0f8274ef992177d97e. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: NON_VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: true } } }
I20260812 11:24:26.199399 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: LEARNER
I20260812 11:24:26.199748 24413 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: attempting to promote NON_VOTER 6dc45d4b35ca438a8a852f366e9aef2e to VOTER
I20260812 11:24:26.200544 24413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } }
I20260812 11:24:26.203042 24024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Refusing update from remote peer 5cde5ff1cbff4d0f8274ef992177d97e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 11:24:26.203243 24328 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 LEARNER]: Refusing update from remote peer 5cde5ff1cbff4d0f8274ef992177d97e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260812 11:24:26.203676 24395 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 11:24:26.204011 24413 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260812 11:24:26.206069 24395 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dc45d4b35ca438a8a852f366e9aef2e (127.12.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.206668 24024 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dc45d4b35ca438a8a852f366e9aef2e (127.12.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.209206 24413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.209285 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.209481 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:26.209491 24413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:26.209692 24395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.209935 24395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:26.210861 24328 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dc45d4b35ca438a8a852f366e9aef2e (127.12.110.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.216569 24371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.216905 24371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:26.217679 24431 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:26.217958 24431 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:26.235327 24279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:26.239703 24279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:26.240015 24279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:26.241083 24279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:26.242775 24279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:26.242903 24279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:26.243040 24279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:26.243139 24279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:26.279484 24279 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:46721
I20260812 11:24:26.279717 24606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:46721 every 8 connection(s)
I20260812 11:24:26.281076 24279 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:26.284384 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 24279
I20260812 11:24:26.284756 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 23366
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.252@KRBTEST.COM
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.194@KRBTEST.COM
I20260812 11:24:26.324314 24419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.284544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:48003 (local address 127.12.110.252:34385)
0812 11:24:26.284782 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.284788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.297257 (+ 12469us) server_negotiation.cc:407] Connection header received
0812 11:24:26.299480 (+ 2223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.299485 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.299561 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.299619 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.301479 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.302268 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.305510 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.306134 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.306234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.315710 (+ 9476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.315718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.315723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.315773 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.319660 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.321026 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.321032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.321036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.321207 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.321802 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.321807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.321810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.322203 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.322351 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.323608 (+ 1257us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.323895 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260812 11:24:26.326959 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.285006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.12.110.193:48003)
0812 11:24:26.297085 (+ 12079us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:26.297129 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:26.299144 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:26.300050 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:26.300058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:26.300714 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:26.301295 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.301312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.302446 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.302449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.305289 (+ 2840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.305310 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.309373 (+ 4063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.309383 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.309844 (+ 461us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.311880 (+ 2036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:26.311918 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:26.315394 (+ 3476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:26.319901 (+ 4507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.319916 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.319937 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.320767 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.321385 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.321389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.321393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.321659 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.322458 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:26.322467 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:26.323076 (+ 609us) client_negotiation.cc:769] Sending connection context
0812 11:24:26.324728 (+ 1652us) client_negotiation.cc:240] Negotiation successful
0812 11:24:26.325358 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11827,"thread_start_us":26327,"threads_started":1}
I20260812 11:24:26.329736 24608 heartbeater.cc:347] Connected to a master server at 127.12.110.252:34385
I20260812 11:24:26.330129 24608 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:26.331454 24608 heartbeater.cc:511] Master 127.12.110.252:34385 requested a full tablet report, sending...
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:26.334507 24313 ts_manager.cc:194] Registered new tserver with Master: bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:26.336544 24622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.287183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.12.110.194:41603)
0812 11:24:26.305855 (+ 18672us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.305864 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.305893 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:24:26.305968 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.305977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.306238 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.306413 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.310439 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.312212 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.314213 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.315102 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.316707 (+ 1605us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.322091 (+ 5384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.322106 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.322126 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.322166 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.326218 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.327229 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.327237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.327240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.327423 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.327919 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.327923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.327926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.328429 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.335595 (+ 7166us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.335786 (+ 191us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.336103 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18498,"thread_start_us":18330,"threads_started":1}
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.253@KRBTEST.COM
I20260812 11:24:26.337906 23569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:26.343856 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.285835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:48709 (local address 127.12.110.254:44299)
0812 11:24:26.318134 (+ 32299us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.318142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.318181 (+ 39us) server_negotiation.cc:407] Connection header received
0812 11:24:26.318289 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.318296 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.318404 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.318689 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.321293 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.322724 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.325446 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.326196 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.326315 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.335099 (+ 8784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.335106 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.335111 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.335152 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.339757 (+ 4605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.340856 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.340887 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.340892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.341096 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.341638 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.341646 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.341650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.342284 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.342440 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.343055 (+ 615us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.343330 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32089,"thread_start_us":34292,"threads_started":1}
I20260812 11:24:26.345499 24613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.285740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:48709)
0812 11:24:26.311891 (+ 26151us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:26.311917 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:26.312087 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:26.319051 (+ 6964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:26.319057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:26.320546 (+ 1489us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:26.321050 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.321064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.322913 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.322918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.325073 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.325107 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.329942 (+ 4835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.329954 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.330281 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.331171 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:26.331196 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:26.334841 (+ 3645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:26.339936 (+ 5095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.339944 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.339950 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.340579 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.341231 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.341236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.341240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.341481 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.342444 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:26.342470 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:26.342670 (+ 200us) client_negotiation.cc:769] Sending connection context
0812 11:24:26.342824 (+ 154us) client_negotiation.cc:240] Negotiation successful
0812 11:24:26.343512 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25950,"thread_start_us":12832,"threads_started":1}
I20260812 11:24:26.346510 24609 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44299
I20260812 11:24:26.346700 24609 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:26.347204 24609 heartbeater.cc:511] Master 127.12.110.254:44299 requested a full tablet report, sending...
I20260812 11:24:26.348076 24616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.285766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.12.110.193:33525)
0812 11:24:26.327953 (+ 42187us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:26.327983 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:26.328226 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:26.328862 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:26.328868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:26.329386 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:26.329601 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.329614 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.331344 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.331349 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.332964 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.332982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.333977 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.333985 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.334144 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.334901 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:26.334922 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:26.338240 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:26.345162 (+ 6922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.345167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.345170 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.345695 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.346291 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.346296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.346300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.346491 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.347069 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:26.347074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:26.347234 (+ 160us) client_negotiation.cc:769] Sending connection context
0812 11:24:26.347377 (+ 143us) client_negotiation.cc:240] Negotiation successful
0812 11:24:26.347718 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42068,"spinlock_wait_cycles":17152,"thread_start_us":45065,"threads_started":1}
I20260812 11:24:26.348467 24621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.286278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:33525 (local address 127.12.110.253:37647)
0812 11:24:26.317155 (+ 30877us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.317165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.328108 (+ 10943us) server_negotiation.cc:407] Connection header received
0812 11:24:26.328394 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.328402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.328513 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.328601 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.329794 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.331150 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.333137 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.333799 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.333926 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.338568 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.338578 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.338586 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.338641 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.345019 (+ 6378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.346047 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.346051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.346053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.346167 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.346659 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.346663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.346665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.346942 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.347099 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.347739 (+ 640us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.347972 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30646,"thread_start_us":30898,"threads_started":1}
I20260812 11:24:26.348701 23037 ts_manager.cc:194] Re-registered known tserver with Master: bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:26.349277 24610 heartbeater.cc:347] Connected to a master server at 127.12.110.253:37647
I20260812 11:24:26.349378 24610 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:26.349745 24610 heartbeater.cc:511] Master 127.12.110.253:37647 requested a full tablet report, sending...
I20260812 11:24:26.349902 23037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:48709
I20260812 11:24:26.351482 24003 ts_manager.cc:194] Registered new tserver with Master: bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:26.427175 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 23366
I20260812 11:24:26.457064 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:41603
--local_ip_for_outbound_sockets=127.12.110.194
--tserver_master_addrs=127.12.110.254:44299
--webserver_port=34473
--webserver_interface=127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab
--principal=oryx/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:26.811410 24624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:26.811853 24624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:26.811990 24624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:26.820865 24624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:26.821009 24624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:26.821141 24624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:26.831728 24624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab
--principal=oryx/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:41603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=34473
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:26.832986 24624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:26.834249 24624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:26.849351 24638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:26.849428 24634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:26.849452 24633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:26.850020 24624 server_base.cc:1034] running on GCE node
I20260812 11:24:26.850759 24624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:26.851682 24624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:26.852921 24624 hybrid_clock.cc:648] HybridClock initialized: now 1786533866852888 us; error 32 us; skew 500 ppm
I20260812 11:24:26.853297 24624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:26.857326 24624 init.cc:377] Logged in from keytab as oryx/127.12.110.194@KRBTEST.COM (short username oryx)
I20260812 11:24:26.859750 24624 webserver.cc:460] Webserver started at http://127.12.110.194:34473/ using document root <none> and password file <none>
I20260812 11:24:26.860558 24624 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:26.860738 24624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:26.866585 24624 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 11:24:26.869304 24649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:26.870895 24624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 11:24:26.871066 24624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "b4470d8736a04cd4b16b8ffda7aba8e5"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "02939659fc0423a8f6a381f0eb35bfea"
server_key_iv: "87c8ade1a9748a05e1833ca39235dd2f"
server_key_version: "encryptionkey@0"
I20260812 11:24:26.872134 24624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:26.884590 24624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:26.888602 24624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:26.888906 24624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:26.889884 24624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:26.891392 24624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:26.891481 24624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:26.891561 24624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:26.891616 24624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:26.926381 24624 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:41603
I20260812 11:24:26.926476 24812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:41603 every 8 connection(s)
I20260812 11:24:26.927897 24624 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:26.937748 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 24624
I20260812 11:24:26.938100 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 23583
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.253@KRBTEST.COM
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533866, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.252@KRBTEST.COM
Aug 12 11:24:26 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.195@KRBTEST.COM
I20260812 11:24:26.966861 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.930967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:57389 (local address 127.12.110.254:44299)
0812 11:24:26.946888 (+ 15921us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.946895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.946918 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:24:26.947000 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.947008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.947095 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.947213 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.949104 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.950232 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.952745 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.953860 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.954007 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.958360 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.958367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.958372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.958414 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.962582 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.963595 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.963602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.963606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.963782 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.964408 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.964415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.964419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.964800 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.964943 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.966094 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.966374 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15791,"thread_start_us":16383,"threads_started":1}
I20260812 11:24:26.968745 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.931331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:40615 (local address 127.12.110.252:34385)
0812 11:24:26.946685 (+ 15354us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.946694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.946713 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:24:26.946777 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.946783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.946868 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.947691 (+ 823us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.949145 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.950257 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.952745 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.953856 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.956379 (+ 2523us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.959602 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.959610 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.959618 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.959659 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.964329 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.965246 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.965253 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.965256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.965401 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.965889 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.965897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.965903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.966298 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.966438 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.968112 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.968350 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15207,"thread_start_us":15077,"threads_started":1}
I20260812 11:24:26.969090 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.931803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.194:57389)
0812 11:24:26.940286 (+ 8483us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:26.940335 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:26.945517 (+ 5182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:26.947296 (+ 1779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:26.947303 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:26.948398 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:26.948926 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.948953 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.950389 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.950394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.952619 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.952634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.954093 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.954097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.954213 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.954860 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:26.954880 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:26.958108 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:26.962758 (+ 4650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.962767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.962786 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.963440 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.963968 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.963972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.963975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.964256 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.965051 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:26.965061 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:26.965597 (+ 536us) client_negotiation.cc:769] Sending connection context
0812 11:24:26.967085 (+ 1488us) client_negotiation.cc:240] Negotiation successful
0812 11:24:26.967621 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8181,"mutex_wait_us":403,"thread_start_us":7931,"threads_started":1}
I20260812 11:24:26.969327 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.930869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:60457 (local address 127.12.110.253:37647)
0812 11:24:26.942588 (+ 11719us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.942595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.942617 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:24:26.945688 (+ 3071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.945697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.945819 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.947508 (+ 1689us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.949141 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.949864 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.952799 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.953209 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.953380 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.961411 (+ 8031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.961422 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.961426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.961474 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.965444 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.966283 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.966288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.966291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.966452 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.966889 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.966893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.966896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.967283 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.967443 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.968663 (+ 1220us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.968903 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11584,"thread_start_us":11534,"threads_started":1}
I20260812 11:24:26.969502 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.932373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.12.110.194:40615)
0812 11:24:26.941726 (+ 9353us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:26.941748 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:26.945032 (+ 3284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:26.947056 (+ 2024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:26.947062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:26.947882 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:26.948948 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.948965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.950433 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.950438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.952560 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.952587 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.954502 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.954781 (+ 279us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.955025 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.956022 (+ 997us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:26.956045 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:26.959383 (+ 3338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:26.964509 (+ 5126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.964516 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.964522 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.965087 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.965549 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.965557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.965560 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.965737 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.966505 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:26.966511 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:26.966692 (+ 181us) client_negotiation.cc:769] Sending connection context
0812 11:24:26.966857 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 11:24:26.967459 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9213,"mutex_wait_us":184,"spinlock_wait_cycles":41344,"thread_start_us":9033,"threads_started":1}
I20260812 11:24:26.969820 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.932758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.12.110.194:60457)
0812 11:24:26.942279 (+ 9521us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:26.942300 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:26.945402 (+ 3102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:26.946229 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:26.946240 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:26.947785 (+ 1545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:26.948926 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.948953 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.950028 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.950033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.952619 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:26.952634 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.953444 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.953454 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:26.953703 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.954571 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:26.954613 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:26.960808 (+ 6195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:26.965585 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.965589 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.965593 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.966137 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.966603 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:26.966607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:26.966611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:26.966746 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:26.967504 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:26.967508 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:26.967664 (+ 156us) client_negotiation.cc:769] Sending connection context
0812 11:24:26.967822 (+ 158us) client_negotiation.cc:240] Negotiation successful
0812 11:24:26.968107 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9400,"mutex_wait_us":332,"spinlock_wait_cycles":4480,"thread_start_us":9278,"threads_started":1}
I20260812 11:24:26.970896 24813 heartbeater.cc:347] Connected to a master server at 127.12.110.252:34385
I20260812 11:24:26.971153 24816 heartbeater.cc:347] Connected to a master server at 127.12.110.253:37647
I20260812 11:24:26.971275 24813 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:26.971304 24816 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:26.971973 24814 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44299
I20260812 11:24:26.972117 24814 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:26.972358 24816 heartbeater.cc:511] Master 127.12.110.253:37647 requested a full tablet report, sending...
I20260812 11:24:26.972613 24814 heartbeater.cc:511] Master 127.12.110.254:44299 requested a full tablet report, sending...
I20260812 11:24:26.973706 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:26.939794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50126 (local address 127.12.110.195:45185)
0812 11:24:26.945403 (+ 5609us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:26.945412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:26.946657 (+ 1245us) server_negotiation.cc:407] Connection header received
0812 11:24:26.946738 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:26.946746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:26.946997 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:26.947147 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:26.951225 (+ 4078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.954212 (+ 2987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.956713 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:26.957677 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:26.957822 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:26.964282 (+ 6460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:26.964301 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:26.964322 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:26.964373 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:26.968750 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.971256 (+ 2506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.971268 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.971273 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.971447 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:26.971861 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:26.971866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:26.971870 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:26.972320 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:26.972647 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:26.973070 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 11:24:26.973334 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5433,"thread_start_us":5257,"threads_started":1}
I20260812 11:24:26.974081 23037 ts_manager.cc:194] Re-registered known tserver with Master: b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603)
I20260812 11:24:26.974349 24003 ts_manager.cc:194] Registered new tserver with Master: b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603)
I20260812 11:24:26.975600 23037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:57389
I20260812 11:24:26.975749 23772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:26.976177 24813 heartbeater.cc:511] Master 127.12.110.252:34385 requested a full tablet report, sending...
I20260812 11:24:26.977334 24313 ts_manager.cc:194] Registered new tserver with Master: b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603)
I20260812 11:24:27.067261 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 23583
I20260812 11:24:27.093556 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:45185
--local_ip_for_outbound_sockets=127.12.110.195
--tserver_master_addrs=127.12.110.254:44299
--webserver_port=36327
--webserver_interface=127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab
--principal=oryx/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:24:27.353566 24609 heartbeater.cc:503] Master 127.12.110.254:44299 was elected leader, sending a full tablet report...
W20260812 11:24:27.435647 24834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:27.435911 24834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:27.436004 24834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:27.443373 24834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:27.443471 24834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:27.443591 24834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:27.453773 24834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab
--principal=oryx/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:45185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=36327
--tserver_master_addrs=127.12.110.254:44299,127.12.110.253:37647,127.12.110.252:34385
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:27.455250 24834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:27.456633 24834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:27.470129 24841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:27.470197 24845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:27.470131 24847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:27.471658 24834 server_base.cc:1034] running on GCE node
I20260812 11:24:27.472268 24834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:27.473186 24834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:27.474421 24834 hybrid_clock.cc:648] HybridClock initialized: now 1786533867474377 us; error 66 us; skew 500 ppm
I20260812 11:24:27.474758 24834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:27 dist-test-slave-h6n0 krb5kdc[22968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533867, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:27.478613 24834 init.cc:377] Logged in from keytab as oryx/127.12.110.195@KRBTEST.COM (short username oryx)
I20260812 11:24:27.480752 24834 webserver.cc:460] Webserver started at http://127.12.110.195:36327/ using document root <none> and password file <none>
I20260812 11:24:27.481671 24834 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:27.481873 24834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:27.488754 24834 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:24:27.492538 24857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:27.494143 24834 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:24:27.494342 24834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "7f70661a5b854a4b85823ad896d6ab12"
format_stamp: "Formatted at 2026-08-12 11:24:23 on dist-test-slave-h6n0"
server_key: "d0c51f923e4b3a3bf344bf8706181428"
server_key_iv: "675ff0ee4989e6cb5a128971f122c820"
server_key_version: "encryptionkey@0"
I20260812 11:24:27.496239 24834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:27.509706 24834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:27.514393 24834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:27.514842 24834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:27.515868 24834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:27.517428 24834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:27.517561 24834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:27.517702 24834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:27.517794 24834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:27.553499 24834 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:45185
I20260812 11:24:27.553589 25054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:45185 every 8 connection(s)
I20260812 11:24:27.555135 24834 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:27.563150 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 24834
Aug 12 11:24:27 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533867, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
Aug 12 11:24:27 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533867, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.253@KRBTEST.COM
Aug 12 11:24:27 dist-test-slave-h6n0 krb5kdc[22968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533867, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.252@KRBTEST.COM
I20260812 11:24:27.615703 25064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.559459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:52629 (local address 127.12.110.252:34385)
0812 11:24:27.573492 (+ 14033us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.573502 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.573527 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:24:27.573601 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.573609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.573710 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.573839 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:27.576464 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.577938 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.580494 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.581121 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.581255 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.602696 (+ 21441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:27.602708 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:27.602717 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:27.602785 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:27.608315 (+ 5530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.609852 (+ 1537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.609861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.609868 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.610079 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.610767 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.610775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.610781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.611287 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:27.611464 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.614784 (+ 3320us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.615148 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13869,"thread_start_us":13732,"threads_started":1}
I20260812 11:24:27.619160 25066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.559966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.12.110.195:52629)
0812 11:24:27.572516 (+ 12550us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:27.572540 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:27.572673 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:27.574077 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:27.574085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:27.575397 (+ 1312us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:27.576244 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.576262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.578285 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.578291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.580307 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.580322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.581334 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.581342 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.581523 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.592883 (+ 11360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:27.592916 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:27.602318 (+ 9402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:27.608583 (+ 6265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:27.608600 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:27.608629 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:27.609643 (+ 1014us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:27.610248 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:27.610256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:27.610262 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:27.610599 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:27.611563 (+ 964us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:27.611577 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:27.612212 (+ 635us) client_negotiation.cc:769] Sending connection context
0812 11:24:27.616446 (+ 4234us) client_negotiation.cc:240] Negotiation successful
0812 11:24:27.617226 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8974,"mutex_wait_us":49,"thread_start_us":8785,"threads_started":1}
I20260812 11:24:27.622110 25063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.559109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.12.110.195:60443)
0812 11:24:27.565253 (+ 6144us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:27.565275 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:27.565421 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:27.575451 (+ 10030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:27.575457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:27.578563 (+ 3106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:27.579631 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.579645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.582072 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.582079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.584095 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.584116 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.585552 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.585561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.585810 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.588809 (+ 2999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:27.588842 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:27.592391 (+ 3549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:27.612071 (+ 19680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:27.612081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:27.612087 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:27.612791 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:27.614112 (+ 1321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:27.614121 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:27.614128 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:27.614358 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:27.620746 (+ 6388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:27.620755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:27.620982 (+ 227us) client_negotiation.cc:769] Sending connection context
0812 11:24:27.621147 (+ 165us) client_negotiation.cc:240] Negotiation successful
0812 11:24:27.621494 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5998,"spinlock_wait_cycles":16768,"thread_start_us":5749,"threads_started":1}
I20260812 11:24:27.622740 25055 heartbeater.cc:347] Connected to a master server at 127.12.110.252:34385
I20260812 11:24:27.623157 25055 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:27.623996 25065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.559445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:60443 (local address 127.12.110.253:37647)
0812 11:24:27.575066 (+ 15621us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.575072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.575094 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:24:27.575156 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.575161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.575253 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.575868 (+ 615us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:27.579818 (+ 3950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.581892 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.584363 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.585282 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.586214 (+ 932us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.592675 (+ 6461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:27.592685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:27.592692 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:27.592747 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:27.611831 (+ 19084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.613026 (+ 1195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.613034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.613041 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.613253 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.614613 (+ 1360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.614621 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.614627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.615089 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:27.622968 (+ 7879us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.623111 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.623552 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15457,"thread_start_us":15287,"threads_started":1}
I20260812 11:24:27.624266 25055 heartbeater.cc:511] Master 127.12.110.252:34385 requested a full tablet report, sending...
I20260812 11:24:27.625168 25058 heartbeater.cc:347] Connected to a master server at 127.12.110.253:37647
I20260812 11:24:27.625299 25058 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:27.625798 25058 heartbeater.cc:511] Master 127.12.110.253:37647 requested a full tablet report, sending...
I20260812 11:24:27.626726 24313 ts_manager.cc:194] Registered new tserver with Master: 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185)
I20260812 11:24:27.627002 24003 ts_manager.cc:194] Registered new tserver with Master: 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185)
I20260812 11:24:27.629801 25061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.558287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:40373 (local address 127.12.110.254:44299)
0812 11:24:27.558950 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.558957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.565269 (+ 6312us) server_negotiation.cc:407] Connection header received
0812 11:24:27.565378 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.565383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.565489 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.565628 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:27.567605 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.568944 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.572343 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.573253 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.573389 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.599125 (+ 25736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:27.599164 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:27.599177 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:27.599276 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:27.620510 (+ 21234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.627486 (+ 6976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.627493 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.627497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.627745 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.628213 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.628218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.628222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.628644 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:27.628798 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.629185 (+ 387us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.629420 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":310,"threads_started":1}
I20260812 11:24:27.629928 25062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.558947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.195:40373)
0812 11:24:27.560034 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:27.560108 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:27.564223 (+ 4115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:27.565862 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:27.565870 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:27.566425 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:27.567414 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.567446 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.569102 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.569109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.572160 (+ 3051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.572183 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.573486 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.573496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.573914 (+ 418us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.575167 (+ 1253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:27.575200 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:27.581091 (+ 5891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:27.620678 (+ 39587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:27.620683 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:27.620686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:27.621233 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:27.627884 (+ 6651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:27.627889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:27.627893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:27.628087 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:27.628759 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:27.628764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:27.628930 (+ 166us) client_negotiation.cc:769] Sending connection context
0812 11:24:27.629077 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 11:24:27.629401 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":2331,"threads_started":1}
I20260812 11:24:27.631963 25057 heartbeater.cc:347] Connected to a master server at 127.12.110.254:44299
I20260812 11:24:27.632128 25057 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:27.632634 25057 heartbeater.cc:511] Master 127.12.110.254:44299 requested a full tablet report, sending...
I20260812 11:24:27.634123 23037 ts_manager.cc:194] Re-registered known tserver with Master: 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185)
I20260812 11:24:27.635404 23037 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:40373
I20260812 11:24:27.644534 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.582231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34730 (local address 127.12.110.252:34385)
0812 11:24:27.596852 (+ 14621us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.596857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.617692 (+ 20835us) server_negotiation.cc:407] Connection header received
0812 11:24:27.617969 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.617977 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.618070 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.618161 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:27.621483 (+ 3322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.623093 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.627596 (+ 4503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.628314 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.628442 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.636095 (+ 7653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:27.636104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:27.636109 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:27.636152 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:27.640000 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.640921 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.640926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.640930 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.641117 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.641561 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.641566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.641569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.641848 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:27.642173 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.642787 (+ 614us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.643018 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14490,"thread_start_us":14348,"threads_started":1}
I20260812 11:24:27.646513 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.582717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.12.110.253:37647)
0812 11:24:27.618365 (+ 35648us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.618373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.618394 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:24:27.618449 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.618453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.618562 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.618685 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:27.624661 (+ 5976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.629304 (+ 4643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.633616 (+ 4312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.634489 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.634670 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.639085 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:27.639090 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:27.639094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:27.639130 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:27.642890 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.643736 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.643744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.643747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.643930 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.644326 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.644333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.644337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.644729 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:27.645606 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.645748 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.646058 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35487,"thread_start_us":37915,"threads_started":1}
I20260812 11:24:27.650585 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.582772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35190 (local address 127.12.110.254:44299)
0812 11:24:27.617913 (+ 35141us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.617923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.627961 (+ 10038us) server_negotiation.cc:407] Connection header received
0812 11:24:27.628025 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.628030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.628149 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.628258 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:27.632262 (+ 4004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.633479 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.635997 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.638821 (+ 2824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.638979 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.643558 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:27.643568 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:27.643573 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:27.643613 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:27.647862 (+ 4249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.648530 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.648535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.648540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.648667 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:27.649067 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:27.649074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:27.649081 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:27.649354 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:27.649636 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.649952 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.650176 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34975,"thread_start_us":40457,"threads_started":1}
I20260812 11:24:27.654847 23037 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35190:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:27.661885 23037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:27.713001 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.686412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.12.110.194:41603)
0812 11:24:27.702662 (+ 16250us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.702677 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.702728 (+ 51us) server_negotiation.cc:407] Connection header received
0812 11:24:27.702906 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.702919 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.703305 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.703851 (+ 546us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:27.706877 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.707964 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.710517 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.711850 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.712042 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.712234 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.712543 (+ 309us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.712711 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15951,"thread_start_us":19015,"threads_started":1}
I20260812 11:24:27.713104 25100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.685885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:41603 (local address 127.0.0.1:45526)
0812 11:24:27.688352 (+ 2467us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:27.688372 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:27.688498 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:27.703834 (+ 15336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:27.703843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:27.703896 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:27.704749 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.704763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.708120 (+ 3357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.708125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.710357 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.710374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.712133 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.712140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.712352 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.712377 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:24:27.712484 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:24:27.712624 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2312,"mutex_wait_us":27,"thread_start_us":2181,"threads_started":1}
I20260812 11:24:27.713904 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.689728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.12.110.193:46721)
0812 11:24:27.705410 (+ 15682us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.705419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.705449 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:24:27.705546 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.705553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.705809 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.706313 (+ 504us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:27.707124 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.708334 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.710171 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.712527 (+ 2356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.712711 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.712850 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.713414 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.713582 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15474,"thread_start_us":15365,"threads_started":1}
I20260812 11:24:27.713905 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.689413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:34418)
0812 11:24:27.705008 (+ 15595us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:27.705037 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:27.705359 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:27.706295 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:27.706300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:27.706328 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:27.706909 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.706932 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.708495 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.708500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.709994 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.710007 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.713036 (+ 3029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.713045 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.713230 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.713259 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:24:27.713368 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 11:24:27.713495 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15378,"mutex_wait_us":2,"spinlock_wait_cycles":4864,"thread_start_us":15093,"threads_started":1}
I20260812 11:24:27.715984 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.688433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45185 (local address 127.0.0.1:50128)
0812 11:24:27.689417 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:27.689445 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:27.689637 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:27.706385 (+ 16748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:27.706390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:27.706429 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:27.706969 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.706979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.710378 (+ 3399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.710383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.713070 (+ 2687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:27.713083 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.715077 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.715084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:27.715294 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.715325 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:24:27.715421 (+ 96us) client_negotiation.cc:240] Negotiation successful
0812 11:24:27.715574 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"spinlock_wait_cycles":896,"thread_start_us":555,"threads_started":1}
I20260812 11:24:27.716091 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:27.688988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50128 (local address 127.12.110.195:45185)
0812 11:24:27.705942 (+ 16954us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:27.705947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:27.705975 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:24:27.706048 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:27.706052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:27.706188 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:27.706392 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:27.708545 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.710226 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.713532 (+ 3306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:27.714872 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:27.715048 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:27.715202 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:27.715527 (+ 325us) server_negotiation.cc:299] Negotiation successful
0812 11:24:27.715692 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16791,"thread_start_us":17552,"threads_started":1}
I20260812 11:24:27.718392 24720 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7d921cf40c43bfb4e72149bfd8ac30 (DEFAULT_TABLE table=test-table [id=168509aa6dce48ba955dc05068359618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:27.718658 24502 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7d921cf40c43bfb4e72149bfd8ac30 (DEFAULT_TABLE table=test-table [id=168509aa6dce48ba955dc05068359618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:27.719962 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7d921cf40c43bfb4e72149bfd8ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:27.719959 24720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7d921cf40c43bfb4e72149bfd8ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:27.719506 24937 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7d921cf40c43bfb4e72149bfd8ac30 (DEFAULT_TABLE table=test-table [id=168509aa6dce48ba955dc05068359618]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:27.720831 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7d921cf40c43bfb4e72149bfd8ac30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:27.733778 25107 tablet_bootstrap.cc:492] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Bootstrap starting.
I20260812 11:24:27.735472 25108 tablet_bootstrap.cc:492] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Bootstrap starting.
I20260812 11:24:27.736439 25109 tablet_bootstrap.cc:492] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Bootstrap starting.
I20260812 11:24:27.739137 25107 tablet_bootstrap.cc:654] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:27.739656 25108 tablet_bootstrap.cc:654] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:27.739753 25109 tablet_bootstrap.cc:654] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:27.740571 25107 log.cc:824] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:27.740777 25109 log.cc:824] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:27.740839 25108 log.cc:824] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:27.743794 25107 tablet_bootstrap.cc:492] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: No bootstrap required, opened a new log
I20260812 11:24:27.744086 25107 ts_tablet_manager.cc:1397] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 11:24:27.748287 25108 tablet_bootstrap.cc:492] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: No bootstrap required, opened a new log
I20260812 11:24:27.748582 25108 ts_tablet_manager.cc:1397] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260812 11:24:27.748826 25109 tablet_bootstrap.cc:492] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: No bootstrap required, opened a new log
I20260812 11:24:27.749092 25109 ts_tablet_manager.cc:1397] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260812 11:24:27.754751 25107 raft_consensus.cc:348] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:27.755225 25107 raft_consensus.cc:374] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:27.755326 25107 raft_consensus.cc:729] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8713ea524e4655b599a1619c945cc4, State: Initialized, Role: FOLLOWER
I20260812 11:24:27.755975 25107 consensus_queue.cc:260] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:27.756209 25108 raft_consensus.cc:348] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:27.756728 25108 raft_consensus.cc:374] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:27.756875 25108 raft_consensus.cc:729] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4470d8736a04cd4b16b8ffda7aba8e5, State: Initialized, Role: FOLLOWER
I20260812 11:24:27.757488 25108 consensus_queue.cc:260] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:27.757738 25107 ts_tablet_manager.cc:1428] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260812 11:24:27.759464 25109 raft_consensus.cc:348] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:27.759915 25109 raft_consensus.cc:374] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:27.760042 25109 raft_consensus.cc:729] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f70661a5b854a4b85823ad896d6ab12, State: Initialized, Role: FOLLOWER
I20260812 11:24:27.760684 25109 consensus_queue.cc:260] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:27.763226 25109 ts_tablet_manager.cc:1428] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260812 11:24:27.763581 25057 heartbeater.cc:503] Master 127.12.110.254:44299 was elected leader, sending a full tablet report...
I20260812 11:24:27.765591 25108 ts_tablet_manager.cc:1428] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Time spent starting tablet: real 0.017s user 0.007s sys 0.008s
I20260812 11:24:27.766880 24814 heartbeater.cc:503] Master 127.12.110.254:44299 was elected leader, sending a full tablet report...
W20260812 11:24:27.785034 24611 tablet.cc:2369] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:24:27.809561 25059 tablet.cc:2369] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:24:27.931610 24818 tablet.cc:2369] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:28.083278 25118 raft_consensus.cc:482] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:28.083794 25118 raft_consensus.cc:504] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:28.086027 25118 leader_election.cc:288] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185), bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:28.093380 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.087553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:45699 (local address 127.12.110.193:46721)
0812 11:24:28.087797 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:28.087805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:28.088193 (+ 388us) server_negotiation.cc:407] Connection header received
0812 11:24:28.088421 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:28.088426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:28.088497 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:28.088593 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:28.089088 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.089794 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.091616 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.092412 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.092595 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.092674 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:28.092970 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 11:24:28.093095 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 11:24:28.093487 25128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.087429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.12.110.194:45699)
0812 11:24:28.087978 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:28.087991 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:28.088272 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:28.088611 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:28.088615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:28.088638 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:28.088942 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:28.088950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.089922 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.089929 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:28.091457 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:28.091470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.092629 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.092635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:28.092786 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.092810 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:24:28.092928 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 11:24:28.093081 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":18,"thread_start_us":303,"threads_started":1}
I20260812 11:24:28.094854 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.086705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:41457 (local address 127.12.110.195:45185)
0812 11:24:28.086934 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:28.086940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:28.087507 (+ 567us) server_negotiation.cc:407] Connection header received
0812 11:24:28.087814 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:28.087820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:28.087911 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:28.087985 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:28.088699 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.089428 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.091792 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.093246 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.093466 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.093552 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:28.094429 (+ 877us) server_negotiation.cc:299] Negotiation successful
0812 11:24:28.094668 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:24:28.094981 25124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.086393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45185 (local address 127.12.110.194:41457)
0812 11:24:28.087365 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:28.087390 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:28.087569 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:28.088188 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:28.088191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:28.088215 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:28.088547 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:28.088554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.089578 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.089583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:28.091469 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:28.091497 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.093908 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.093922 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:28.094218 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.094246 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:24:28.094343 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 11:24:28.094549 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":667,"mutex_wait_us":48,"thread_start_us":441,"threads_started":1}
I20260812 11:24:28.095095 24537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7d921cf40c43bfb4e72149bfd8ac30" candidate_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8713ea524e4655b599a1619c945cc4" is_pre_election: true
I20260812 11:24:28.095789 24537 raft_consensus.cc:2454] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4470d8736a04cd4b16b8ffda7aba8e5 in term 0.
I20260812 11:24:28.096082 24973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7d921cf40c43bfb4e72149bfd8ac30" candidate_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f70661a5b854a4b85823ad896d6ab12" is_pre_election: true
I20260812 11:24:28.096563 24973 raft_consensus.cc:2454] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4470d8736a04cd4b16b8ffda7aba8e5 in term 0.
I20260812 11:24:28.096565 24651 leader_election.cc:302] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4470d8736a04cd4b16b8ffda7aba8e5, bc8713ea524e4655b599a1619c945cc4; no voters:
I20260812 11:24:28.097051 25118 raft_consensus.cc:2781] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:28.097241 25118 raft_consensus.cc:482] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:28.097389 25118 raft_consensus.cc:3037] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:28.099792 25118 raft_consensus.cc:504] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:28.100557 25118 leader_election.cc:288] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [CANDIDATE]: Term 1 election: Requested vote from peers 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185), bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:28.101048 24973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7d921cf40c43bfb4e72149bfd8ac30" candidate_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f70661a5b854a4b85823ad896d6ab12"
I20260812 11:24:28.101152 24537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7d921cf40c43bfb4e72149bfd8ac30" candidate_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8713ea524e4655b599a1619c945cc4"
I20260812 11:24:28.101246 24973 raft_consensus.cc:3037] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:28.101430 24537 raft_consensus.cc:3037] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:28.103767 24973 raft_consensus.cc:2454] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4470d8736a04cd4b16b8ffda7aba8e5 in term 1.
I20260812 11:24:28.103886 24537 raft_consensus.cc:2454] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4470d8736a04cd4b16b8ffda7aba8e5 in term 1.
I20260812 11:24:28.104218 24651 leader_election.cc:302] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f70661a5b854a4b85823ad896d6ab12, b4470d8736a04cd4b16b8ffda7aba8e5; no voters:
I20260812 11:24:28.104507 25118 raft_consensus.cc:2781] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:28.105089 25118 raft_consensus.cc:686] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 LEADER]: Becoming Leader. State: Replica: b4470d8736a04cd4b16b8ffda7aba8e5, State: Running, Role: LEADER
I20260812 11:24:28.105692 25118 consensus_queue.cc:237] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } }
I20260812 11:24:28.111030 23035 catalog_manager.cc:5495] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194). New cstate: current_term: 1 leader_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } health_report { overall_health: UNKNOWN } } }
I20260812 11:24:28.182183 25101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.176092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.12.110.194:41603)
0812 11:24:28.176560 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:28.176569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:28.176595 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:24:28.176714 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:28.176718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:28.176800 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:28.176963 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:28.177400 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.178132 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.179982 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.180413 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.180555 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.181036 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:28.181534 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:28.181706 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:28.181766 (+ 60us) server_negotiation.cc:299] Negotiation successful
0812 11:24:28.181862 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260812 11:24:28.190824 24973 raft_consensus.cc:1261] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Refusing update from remote peer b4470d8736a04cd4b16b8ffda7aba8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:28.190891 24537 raft_consensus.cc:1261] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Refusing update from remote peer b4470d8736a04cd4b16b8ffda7aba8e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:28.192003 25129 consensus_queue.cc:1035] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:28.192463 25118 consensus_queue.cc:1035] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:28.209502 25138 mvcc.cc:205] Tried to move back new op lower bound from 7317642724098891776 to 7317642723765059584. Current Snapshot: MvccSnapshot[applied={T|T < 7317642724098891776}]
I20260812 11:24:28.225387 23037 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:28.225720 23037 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35190:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:28.234603 23037 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cde5ff1cbff4d0f8274ef992177d97e: Sending DeleteTablet for 3 replicas of tablet 4e7d921cf40c43bfb4e72149bfd8ac30
I20260812 11:24:28.236243 24720 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7d921cf40c43bfb4e72149bfd8ac30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:28 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:45526
I20260812 11:24:28.236688 24502 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7d921cf40c43bfb4e72149bfd8ac30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:28 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:34418
I20260812 11:24:28.237007 24937 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7d921cf40c43bfb4e72149bfd8ac30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:28 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:50128
I20260812 11:24:28.239147 25154 tablet_replica.cc:330] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: stopping tablet replica
I20260812 11:24:28.239727 25154 raft_consensus.cc:2229] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:28.240415 25154 raft_consensus.cc:2258] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:28.241369 25152 tablet_replica.cc:330] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: stopping tablet replica
I20260812 11:24:28.242967 25152 raft_consensus.cc:2229] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:28.244489 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 22988
I20260812 11:24:28.245015 25154 ts_tablet_manager.cc:1905] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:28.245692 25152 raft_consensus.cc:2258] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:28.245836 23087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:28.246667 25151 tablet_replica.cc:330] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: stopping tablet replica
I20260812 11:24:28.247030 25151 raft_consensus.cc:2229] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:28.247551 25151 raft_consensus.cc:2258] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:28.248957 25152 ts_tablet_manager.cc:1905] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:28.252403 25151 ts_tablet_manager.cc:1905] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:28.257030 25152 ts_tablet_manager.cc:1918] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:28.257236 25152 log.cc:1196] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/4e7d921cf40c43bfb4e72149bfd8ac30
I20260812 11:24:28.257947 25152 ts_tablet_manager.cc:1939] T 4e7d921cf40c43bfb4e72149bfd8ac30 P bc8713ea524e4655b599a1619c945cc4: Deleting consensus metadata
I20260812 11:24:28.262997 25151 ts_tablet_manager.cc:1918] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:28.265007 25151 log.cc:1196] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/4e7d921cf40c43bfb4e72149bfd8ac30
I20260812 11:24:28.265677 25151 ts_tablet_manager.cc:1939] T 4e7d921cf40c43bfb4e72149bfd8ac30 P b4470d8736a04cd4b16b8ffda7aba8e5: Deleting consensus metadata
I20260812 11:24:28.267002 25154 ts_tablet_manager.cc:1918] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:28.267248 25154 log.cc:1196] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/4e7d921cf40c43bfb4e72149bfd8ac30
I20260812 11:24:28.268034 25154 ts_tablet_manager.cc:1939] T 4e7d921cf40c43bfb4e72149bfd8ac30 P 7f70661a5b854a4b85823ad896d6ab12: Deleting consensus metadata
I20260812 11:24:28.288748 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.239478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40954 (local address 127.12.110.253:37647)
0812 11:24:28.258993 (+ 19515us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:28.259002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:28.259045 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:24:28.259146 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:28.259153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:28.259449 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:28.259636 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:28.261632 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.262969 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.264769 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.268395 (+ 3626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.268535 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.279438 (+ 10903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:28.279447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:28.279451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:28.279507 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:28.284620 (+ 5113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.285582 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.285588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.285592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.285735 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.286263 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.286269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.286272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.286730 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:28.286992 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:28.288147 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0812 11:24:28.288380 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19173,"thread_start_us":20286,"threads_started":1}
I20260812 11:24:28.313225 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.243591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.12.110.252:34385)
0812 11:24:28.245557 (+ 1966us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:28.245563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:28.289381 (+ 43818us) server_negotiation.cc:407] Connection header received
0812 11:24:28.289517 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:28.289523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:28.289618 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:28.289734 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:28.294510 (+ 4776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.297047 (+ 2537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.300122 (+ 3075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.301195 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.301303 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.305272 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:28.305279 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:28.305284 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:28.305347 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:28.309007 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.310098 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.310107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.310111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.310543 (+ 432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.311258 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.311265 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.311269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.311699 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:28.311949 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:28.312573 (+ 624us) server_negotiation.cc:299] Negotiation successful
0812 11:24:28.312829 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1840,"thread_start_us":1699,"threads_started":1}
I20260812 11:24:28.427233 23017 catalog_manager.cc:4841] TS bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721): tablet 4e7d921cf40c43bfb4e72149bfd8ac30 (table test-table [id=168509aa6dce48ba955dc05068359618]) successfully deleted
W20260812 11:24:28.427575 23017 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.12.110.193:46721, user_credentials={real_user=slave}} blocked reactor thread for 167560us
I20260812 11:24:28.428006 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 22988
W20260812 11:24:28.473850 24654 connection.cc:535] server connection from 127.0.0.1:45526 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:24:28.509296 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.481105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34744 (local address 127.12.110.252:34385)
0812 11:24:28.481371 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:28.481378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:28.488723 (+ 7345us) server_negotiation.cc:407] Connection header received
0812 11:24:28.489212 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:28.489218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:28.489308 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:28.489393 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:28.492863 (+ 3470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.494038 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.496490 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.497136 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.497271 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.501668 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:28.501674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:28.501677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:28.501711 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:28.505389 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.506256 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.506262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.506265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.506408 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.506870 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.506875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.506878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.507262 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:28.507542 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:28.508555 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0812 11:24:28.508782 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260812 11:24:28.509665 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:28.481056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.12.110.253:37647)
0812 11:24:28.481329 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:28.481340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:28.488738 (+ 7398us) server_negotiation.cc:407] Connection header received
0812 11:24:28.489128 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:28.489137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:28.489242 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:28.489326 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:28.493240 (+ 3914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.494361 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.497123 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:28.497728 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:28.498103 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:28.502299 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:28.502309 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:28.502324 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:28.502373 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:28.506066 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.506956 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.506965 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.506968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.507105 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:28.507562 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:28.507569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:28.507572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:28.507871 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:28.508040 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:28.509077 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0812 11:24:28.509307 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 11:24:29.214712 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.212643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.194:39663)
0812 11:24:29.214342 (+ 1699us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.214474 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1574,"thread_start_us":1457,"threads_started":1}
W20260812 11:24:29.215174 24814 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:44299 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
I20260812 11:24:29.221387 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.220438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:33065)
0812 11:24:29.220977 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.221138 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":265,"threads_started":1}
W20260812 11:24:29.221793 24609 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:44299 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
I20260812 11:24:29.227634 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.226909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.195:60631)
0812 11:24:29.227363 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.227481 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":233,"threads_started":1}
W20260812 11:24:29.228045 25057 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:44299 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
I20260812 11:24:29.734961 25179 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5cde5ff1cbff4d0f8274ef992177d97e)
I20260812 11:24:29.735229 25180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5cde5ff1cbff4d0f8274ef992177d97e)
I20260812 11:24:29.735312 25179 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } }
I20260812 11:24:29.735499 25180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } }
I20260812 11:24:29.736855 25179 leader_election.cc:288] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cde5ff1cbff4d0f8274ef992177d97e (127.12.110.254:44299), 6dc45d4b35ca438a8a852f366e9aef2e (127.12.110.252:34385)
I20260812 11:24:29.749202 25180 leader_election.cc:288] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5cde5ff1cbff4d0f8274ef992177d97e (127.12.110.254:44299), d0e3d3fc75d5404a88526c3e5a7721a3 (127.12.110.253:37647)
I20260812 11:24:29.749773 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.736953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38446)
0812 11:24:29.749237 (+ 12284us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.749437 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12099,"threads_started":1}
I20260812 11:24:29.753567 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.736956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38462)
0812 11:24:29.752985 (+ 16029us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.753243 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15866,"thread_start_us":16299,"threads_started":1}
W20260812 11:24:29.754009 23981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
W20260812 11:24:29.756645 24285 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
I20260812 11:24:29.757211 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.756754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38476)
0812 11:24:29.756915 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.757010 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260812 11:24:29.757437 23981 leader_election.cc:334] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5cde5ff1cbff4d0f8274ef992177d97e (127.12.110.254:44299): Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
I20260812 11:24:29.758908 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.758146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38490)
0812 11:24:29.758357 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.758639 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 11:24:29.759131 24285 leader_election.cc:334] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5cde5ff1cbff4d0f8274ef992177d97e (127.12.110.254:44299): Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
I20260812 11:24:29.762465 25185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.737298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48886 (local address 127.12.110.253:37647)
0812 11:24:29.755632 (+ 18334us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.755638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.755656 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:24:29.755719 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.755724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.755794 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.755935 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.756654 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.757951 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.760505 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.761516 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.761693 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.761772 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.762099 (+ 327us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.762221 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18192,"thread_start_us":17998,"threads_started":1}
I20260812 11:24:29.762568 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.736968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:37647 (local address 127.0.0.1:48886)
0812 11:24:29.751323 (+ 14355us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.751348 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.751524 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.756088 (+ 4564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.756095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.756128 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.756427 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.756443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.758102 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.758110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.760314 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.760327 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.761716 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.761723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.761907 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.761933 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.762049 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.762210 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14229,"thread_start_us":11918,"threads_started":1}
I20260812 11:24:29.763242 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" is_pre_election: true
I20260812 11:24:29.763662 24024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6dc45d4b35ca438a8a852f366e9aef2e in term 1.
I20260812 11:24:29.764217 24281 leader_election.cc:302] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dc45d4b35ca438a8a852f366e9aef2e, d0e3d3fc75d5404a88526c3e5a7721a3; no voters: 5cde5ff1cbff4d0f8274ef992177d97e
I20260812 11:24:29.764535 25184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.737362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:34385 (local address 127.0.0.1:41936)
0812 11:24:29.753502 (+ 16140us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.753522 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.753660 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.757784 (+ 4124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.757790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.757832 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.758243 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.758257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.759727 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.759732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.761838 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.761852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.763652 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.763658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.763834 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.763870 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.763995 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.764127 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16009,"thread_start_us":16343,"threads_started":1}
I20260812 11:24:29.764683 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.755761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.12.110.252:34385)
0812 11:24:29.756668 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.756676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.756695 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:24:29.756755 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.756761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.757622 (+ 861us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.757736 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.758464 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.759581 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.762040 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.763342 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.763547 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.763616 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.764021 (+ 405us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.764154 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":656,"threads_started":1}
I20260812 11:24:29.764746 25180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260812 11:24:29.764912 25180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5cde5ff1cbff4d0f8274ef992177d97e)
I20260812 11:24:29.765067 25180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:24:29.765385 24328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" is_pre_election: true
I20260812 11:24:29.767535 25180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } }
I20260812 11:24:29.768210 25180 leader_election.cc:288] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [CANDIDATE]: Term 2 election: Requested vote from peers 5cde5ff1cbff4d0f8274ef992177d97e (127.12.110.254:44299), d0e3d3fc75d5404a88526c3e5a7721a3 (127.12.110.253:37647)
I20260812 11:24:29.768206 24328 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d0e3d3fc75d5404a88526c3e5a7721a3 in current term 2: Already voted for candidate 6dc45d4b35ca438a8a852f366e9aef2e in this term.
I20260812 11:24:29.768723 24024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3"
I20260812 11:24:29.768872 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.768332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38492)
0812 11:24:29.768617 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.768700 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260812 11:24:29.769012 24024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:24:29.769307 23980 leader_election.cc:302] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d0e3d3fc75d5404a88526c3e5a7721a3; no voters: 5cde5ff1cbff4d0f8274ef992177d97e, 6dc45d4b35ca438a8a852f366e9aef2e
I20260812 11:24:29.770115 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.769628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38498)
0812 11:24:29.769864 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.769956 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260812 11:24:29.770300 24285 leader_election.cc:334] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5cde5ff1cbff4d0f8274ef992177d97e (127.12.110.254:44299): Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
I20260812 11:24:29.771642 24024 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6dc45d4b35ca438a8a852f366e9aef2e in term 2.
I20260812 11:24:29.771818 25179 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:24:29.772127 24281 leader_election.cc:302] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6dc45d4b35ca438a8a852f366e9aef2e, d0e3d3fc75d5404a88526c3e5a7721a3; no voters: 5cde5ff1cbff4d0f8274ef992177d97e
I20260812 11:24:29.772429 25180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:24:29.772862 25180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [term 2 LEADER]: Becoming Leader. State: Replica: 6dc45d4b35ca438a8a852f366e9aef2e, State: Running, Role: LEADER
I20260812 11:24:29.773278 25180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } }
I20260812 11:24:29.775084 25196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dc45d4b35ca438a8a852f366e9aef2e. Latest consensus state: current_term: 2 leader_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:29.775502 25196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:29.776326 25199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:29.780102 25199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:29.780866 25199 catalog_manager.cc:1259] Loaded cluster ID: 490a3b38963f4c17bd8d048ac6a3c817
I20260812 11:24:29.780937 25199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:29.782017 25199 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:29.782871 25199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Loaded TSK: 0
I20260812 11:24:29.783636 25199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:24:29.812333 24309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:29.819854 24309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:29.824025 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.823532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38524)
0812 11:24:29.823741 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.823843 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 11:24:29.824106 24024 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [term 2 FOLLOWER]: Refusing update from remote peer 6dc45d4b35ca438a8a852f366e9aef2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260812 11:24:29.825181 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.824729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38536)
0812 11:24:29.824925 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.825031 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 11:24:29.825088 25196 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260812 11:24:29.825657 24285 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e -> Peer 5cde5ff1cbff4d0f8274ef992177d97e (127.12.110.254:44299): Couldn't send request to peer 5cde5ff1cbff4d0f8274ef992177d97e. Status: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260812 11:24:29.828552 25179 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6dc45d4b35ca438a8a852f366e9aef2e. Latest consensus state: current_term: 2 leader_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:29.828863 25179 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:29.830650 25180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:29.831125 25180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:29.832108 25179 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5cde5ff1cbff4d0f8274ef992177d97e" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 44299 } } peers { permanent_uuid: "d0e3d3fc75d5404a88526c3e5a7721a3" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 37647 } attrs { promote: false } } peers { permanent_uuid: "6dc45d4b35ca438a8a852f366e9aef2e" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 34385 } attrs { promote: false } } }
I20260812 11:24:29.832438 25179 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e3d3fc75d5404a88526c3e5a7721a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:24:29.832697 25203 mvcc.cc:205] Tried to move back new op lower bound from 7317642730792087552 to 7317642730594967552. Current Snapshot: MvccSnapshot[applied={T|T < 7317642730792087552}]
I20260812 11:24:29.850819 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.840882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45185 (local address 127.0.0.1:40740)
0812 11:24:29.844391 (+ 3509us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.844413 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.844563 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.844966 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.844971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.844996 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.845156 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.845165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.846720 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.846725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.848417 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.848427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.849897 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.849903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.850085 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.850121 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.850202 (+ 81us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.850352 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3362,"mutex_wait_us":191,"thread_start_us":1788,"threads_started":1}
I20260812 11:24:29.851347 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.839998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:41603 (local address 127.0.0.1:57204)
0812 11:24:29.840404 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.840425 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.840542 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.844032 (+ 3490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.844036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.844060 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.844398 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.844412 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.846065 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.846070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.848238 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.848254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.850415 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.850421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.850617 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.850649 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.850729 (+ 80us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.850890 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":23}
I20260812 11:24:29.851676 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.840266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39446)
0812 11:24:29.842974 (+ 2708us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.842990 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.843123 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.845294 (+ 2171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.845298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.845321 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.845655 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.845663 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.847132 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.847138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.849067 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.849078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.850848 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.850859 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.851035 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.851069 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.851154 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.851289 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"mutex_wait_us":72}
I20260812 11:24:29.851910 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.840562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.12.110.193:46721)
0812 11:24:29.845023 (+ 4461us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.845027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.845042 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:24:29.845098 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.845101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.845158 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.845277 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.845821 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.846970 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.849217 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.850186 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.850367 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.850440 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.851441 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.851589 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4355,"thread_start_us":4251,"threads_started":1}
I20260812 11:24:29.852245 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.840098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.12.110.194:41603)
0812 11:24:29.843597 (+ 3499us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.843606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.843631 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:24:29.843718 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.843727 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.843840 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.843975 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.844615 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.845913 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.848399 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.849673 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.851651 (+ 1978us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.851721 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.851775 (+ 54us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.851911 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3370,"thread_start_us":3262,"threads_started":1}
I20260812 11:24:29.852520 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.841366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.12.110.195:45185)
0812 11:24:29.841868 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.841876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.844515 (+ 2639us) server_negotiation.cc:407] Connection header received
0812 11:24:29.844715 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.844723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.844800 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.845292 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.845470 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.846520 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.848542 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.849719 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.851313 (+ 1594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.851410 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.851658 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.852234 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":307,"threads_started":1}
I20260812 11:24:29.852632 24502 tablet_service.cc:1467] Processing CreateTablet for tablet a4193cc5f1e44ff38ad52e6fdb06d730 (DEFAULT_TABLE table=test-table [id=6c3c09c4555e4aa8a566c62a4462f273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:29.852926 24720 tablet_service.cc:1467] Processing CreateTablet for tablet a4193cc5f1e44ff38ad52e6fdb06d730 (DEFAULT_TABLE table=test-table [id=6c3c09c4555e4aa8a566c62a4462f273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:29.853113 24502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4193cc5f1e44ff38ad52e6fdb06d730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:29.853183 24937 tablet_service.cc:1467] Processing CreateTablet for tablet a4193cc5f1e44ff38ad52e6fdb06d730 (DEFAULT_TABLE table=test-table [id=6c3c09c4555e4aa8a566c62a4462f273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:29.853348 24720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4193cc5f1e44ff38ad52e6fdb06d730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:29.853520 24937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4193cc5f1e44ff38ad52e6fdb06d730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:29.860731 25216 tablet_bootstrap.cc:492] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: Bootstrap starting.
I20260812 11:24:29.862706 25216 tablet_bootstrap.cc:654] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:29.865569 25220 tablet_bootstrap.cc:492] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: Bootstrap starting.
I20260812 11:24:29.867997 25220 tablet_bootstrap.cc:654] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:29.870338 25218 tablet_bootstrap.cc:492] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: Bootstrap starting.
I20260812 11:24:29.871165 25216 tablet_bootstrap.cc:492] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: No bootstrap required, opened a new log
I20260812 11:24:29.871332 25216 ts_tablet_manager.cc:1397] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.009s
I20260812 11:24:29.872172 25218 tablet_bootstrap.cc:654] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:29.872090 25216 raft_consensus.cc:348] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.872339 25216 raft_consensus.cc:374] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:29.872458 25216 raft_consensus.cc:729] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f70661a5b854a4b85823ad896d6ab12, State: Initialized, Role: FOLLOWER
I20260812 11:24:29.872720 25216 consensus_queue.cc:260] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.874229 25220 tablet_bootstrap.cc:492] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: No bootstrap required, opened a new log
I20260812 11:24:29.874383 25220 ts_tablet_manager.cc:1397] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260812 11:24:29.875308 25220 raft_consensus.cc:348] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.875562 25220 raft_consensus.cc:374] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:29.875667 25220 raft_consensus.cc:729] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc8713ea524e4655b599a1619c945cc4, State: Initialized, Role: FOLLOWER
I20260812 11:24:29.875963 25220 consensus_queue.cc:260] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.876471 25216 ts_tablet_manager.cc:1428] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260812 11:24:29.877360 25055 heartbeater.cc:503] Master 127.12.110.252:34385 was elected leader, sending a full tablet report...
I20260812 11:24:29.881438 25218 tablet_bootstrap.cc:492] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: No bootstrap required, opened a new log
I20260812 11:24:29.881695 25218 ts_tablet_manager.cc:1397] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260812 11:24:29.882064 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.880975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:57075)
0812 11:24:29.881561 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.881778 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"threads_started":1}
I20260812 11:24:29.883014 25218 raft_consensus.cc:348] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.883420 25218 raft_consensus.cc:374] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:29.883553 25218 raft_consensus.cc:729] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4470d8736a04cd4b16b8ffda7aba8e5, State: Initialized, Role: FOLLOWER
I20260812 11:24:29.883687 25220 ts_tablet_manager.cc:1428] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260812 11:24:29.883906 24608 heartbeater.cc:503] Master 127.12.110.252:34385 was elected leader, sending a full tablet report...
I20260812 11:24:29.883898 25218 consensus_queue.cc:260] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.884634 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.884177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:33781)
0812 11:24:29.884394 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.884490 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260812 11:24:29.886722 25218 ts_tablet_manager.cc:1428] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260812 11:24:29.887735 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.873894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.195:58731)
0812 11:24:29.887229 (+ 13335us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.887390 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13188,"threads_started":1}
I20260812 11:24:29.889012 24813 heartbeater.cc:503] Master 127.12.110.252:34385 was elected leader, sending a full tablet report...
I20260812 11:24:29.889199 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.887253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.194:40681)
0812 11:24:29.888721 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.888914 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":1217,"threads_started":1}
I20260812 11:24:29.889810 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.888664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.195:54971)
0812 11:24:29.889578 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.889672 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":811}
I20260812 11:24:29.890369 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.889811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.194:37415)
0812 11:24:29.890134 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.890216 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260812 11:24:29.975690 25229 raft_consensus.cc:482] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:29.976004 25229 raft_consensus.cc:504] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.977705 25225 raft_consensus.cc:482] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:29.977840 25229 leader_election.cc:288] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603), 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185)
I20260812 11:24:29.977957 25225 raft_consensus.cc:504] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.980151 25225 leader_election.cc:288] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603), bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:29.987269 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.978906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:47961 (local address 127.12.110.195:45185)
0812 11:24:29.979104 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.979110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.979340 (+ 230us) server_negotiation.cc:407] Connection header received
0812 11:24:29.979542 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.979547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.979642 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.979726 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.980348 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.981393 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.984453 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.985888 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.986102 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.986188 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.986789 (+ 601us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.986941 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":63}
I20260812 11:24:29.988214 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.978508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45185 (local address 127.12.110.193:47961)
0812 11:24:29.979214 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.979234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.979394 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.979758 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.979763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.979788 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.980146 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.980159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.981565 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.981572 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.984253 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.984270 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.986370 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.986375 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.986603 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.986637 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.987530 (+ 893us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.987696 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":28,"thread_start_us":313,"threads_started":1}
I20260812 11:24:29.988906 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.978286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:50779 (local address 127.12.110.194:41603)
0812 11:24:29.978498 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.978504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.978554 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:24:29.978755 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.978763 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.978868 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.979085 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.979702 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.980760 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.984862 (+ 4102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.986670 (+ 1808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.986850 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.986957 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.988018 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.988610 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:24:29.989667 24973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "bc8713ea524e4655b599a1619c945cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f70661a5b854a4b85823ad896d6ab12" is_pre_election: true
I20260812 11:24:29.989943 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.980743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:60159 (local address 127.12.110.193:46721)
0812 11:24:29.980922 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.980927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.981808 (+ 881us) server_negotiation.cc:407] Connection header received
0812 11:24:29.982049 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.982054 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.982127 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.982193 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.982958 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.985098 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.987664 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.988932 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.989026 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.989081 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.989538 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.989663 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":996,"server-negotiator.queue_time_us":52}
I20260812 11:24:29.990171 25245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.980628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.12.110.195:60159)
0812 11:24:29.981708 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.981728 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.981907 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.982376 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.982382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.982410 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.982769 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.982782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.985266 (+ 2484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.985271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.987471 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.987486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.989224 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.989228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.989371 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.989393 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.989510 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.989635 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"thread_start_us":812,"threads_started":1}
I20260812 11:24:29.990137 24973 raft_consensus.cc:2454] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc8713ea524e4655b599a1619c945cc4 in term 0.
I20260812 11:24:29.990243 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.978139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:41603 (local address 127.12.110.193:50779)
0812 11:24:29.978366 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.978388 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.978563 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.979102 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.979108 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.979161 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.979525 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.979541 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.981010 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.981016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.984496 (+ 3480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.984528 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.987176 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.987184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.987754 (+ 570us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.987801 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.987913 (+ 112us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.989580 (+ 1667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":1536}
I20260812 11:24:29.990825 24424 leader_election.cc:302] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f70661a5b854a4b85823ad896d6ab12, bc8713ea524e4655b599a1619c945cc4; no voters:
I20260812 11:24:29.990984 24537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "7f70661a5b854a4b85823ad896d6ab12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8713ea524e4655b599a1619c945cc4" is_pre_election: true
I20260812 11:24:29.991236 24537 raft_consensus.cc:2454] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f70661a5b854a4b85823ad896d6ab12 in term 0.
I20260812 11:24:29.991365 25229 raft_consensus.cc:2781] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:29.991501 25229 raft_consensus.cc:482] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:29.991245 24750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "bc8713ea524e4655b599a1619c945cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" is_pre_election: true
I20260812 11:24:29.991629 25229 raft_consensus.cc:3037] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:29.991775 24750 raft_consensus.cc:2454] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc8713ea524e4655b599a1619c945cc4 in term 0.
I20260812 11:24:29.991797 24859 leader_election.cc:302] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f70661a5b854a4b85823ad896d6ab12, bc8713ea524e4655b599a1619c945cc4; no voters:
I20260812 11:24:29.992203 25225 raft_consensus.cc:2781] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:29.992327 25225 raft_consensus.cc:482] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:29.992422 25225 raft_consensus.cc:3037] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:29.993773 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.980290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:41603 (local address 127.12.110.195:44839)
0812 11:24:29.981114 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:29.981132 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:29.981233 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:29.984479 (+ 3246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:29.984491 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:29.984605 (+ 114us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:29.985072 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.985089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.986987 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.986993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.989316 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:29.989337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.992957 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.992966 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:29.993169 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.993196 (+ 27us) client_negotiation.cc:769] Sending connection context
0812 11:24:29.993317 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 11:24:29.993486 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714}
I20260812 11:24:29.993813 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:29.980959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:44839 (local address 127.12.110.194:41603)
0812 11:24:29.984050 (+ 3091us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:29.984057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:29.984079 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:24:29.984134 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:29.984139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:29.984204 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:29.984317 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:29.985349 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.986821 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.990651 (+ 3830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:29.992667 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:29.992952 (+ 285us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:29.993036 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:29.993342 (+ 306us) server_negotiation.cc:299] Negotiation successful
0812 11:24:29.993480 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2955,"thread_start_us":2792,"threads_started":1}
I20260812 11:24:29.995057 25229 raft_consensus.cc:504] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.995613 25225 raft_consensus.cc:504] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:29.995965 25229 leader_election.cc:288] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [CANDIDATE]: Term 1 election: Requested vote from peers b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603), 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185)
I20260812 11:24:29.996268 25225 leader_election.cc:288] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [CANDIDATE]: Term 1 election: Requested vote from peers b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603), bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721)
I20260812 11:24:29.996778 24750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "bc8713ea524e4655b599a1619c945cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5"
I20260812 11:24:29.997058 24750 raft_consensus.cc:3037] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:29.997030 24973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "bc8713ea524e4655b599a1619c945cc4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f70661a5b854a4b85823ad896d6ab12"
I20260812 11:24:29.997318 24973 raft_consensus.cc:2379] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bc8713ea524e4655b599a1619c945cc4 in current term 1: Already voted for candidate 7f70661a5b854a4b85823ad896d6ab12 in this term.
I20260812 11:24:29.997327 24537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "7f70661a5b854a4b85823ad896d6ab12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc8713ea524e4655b599a1619c945cc4"
I20260812 11:24:29.997922 24537 raft_consensus.cc:2379] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7f70661a5b854a4b85823ad896d6ab12 in current term 1: Already voted for candidate bc8713ea524e4655b599a1619c945cc4 in this term.
I20260812 11:24:29.997807 24749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "7f70661a5b854a4b85823ad896d6ab12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" is_pre_election: true
I20260812 11:24:29.998653 24748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4193cc5f1e44ff38ad52e6fdb06d730" candidate_uuid: "7f70661a5b854a4b85823ad896d6ab12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5"
I20260812 11:24:29.999500 24859 leader_election.cc:302] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7f70661a5b854a4b85823ad896d6ab12; no voters: b4470d8736a04cd4b16b8ffda7aba8e5, bc8713ea524e4655b599a1619c945cc4
I20260812 11:24:29.999833 25225 raft_consensus.cc:2726] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260812 11:24:30.000288 24750 raft_consensus.cc:2454] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc8713ea524e4655b599a1619c945cc4 in term 1.
I20260812 11:24:30.000715 24424 leader_election.cc:302] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b4470d8736a04cd4b16b8ffda7aba8e5, bc8713ea524e4655b599a1619c945cc4; no voters: 7f70661a5b854a4b85823ad896d6ab12
I20260812 11:24:30.001000 25229 raft_consensus.cc:2781] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:30.001541 25229 raft_consensus.cc:686] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 1 LEADER]: Becoming Leader. State: Replica: bc8713ea524e4655b599a1619c945cc4, State: Running, Role: LEADER
I20260812 11:24:30.002815 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.002354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:40081)
0812 11:24:30.002572 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.002664 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 11:24:30.002861 25229 consensus_queue.cc:237] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } }
I20260812 11:24:30.006373 24309 catalog_manager.cc:5495] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc8713ea524e4655b599a1619c945cc4 (127.12.110.193). New cstate: current_term: 1 leader_uuid: "bc8713ea524e4655b599a1619c945cc4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc8713ea524e4655b599a1619c945cc4" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 46721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 } health_report { overall_health: UNKNOWN } } }
I20260812 11:24:30.053462 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.046739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39450 (local address 127.12.110.193:46721)
0812 11:24:30.046951 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:30.046958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:30.047046 (+ 88us) server_negotiation.cc:407] Connection header received
0812 11:24:30.047239 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:30.047246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:30.047385 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:30.047560 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:30.047933 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.049210 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.050783 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.051450 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.051594 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:30.051974 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:30.052704 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:30.052849 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:30.052990 (+ 141us) server_negotiation.cc:299] Negotiation successful
0812 11:24:30.053081 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 11:24:30.057765 24973 raft_consensus.cc:1261] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Refusing update from remote peer bc8713ea524e4655b599a1619c945cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:30.058072 24750 raft_consensus.cc:1261] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 FOLLOWER]: Refusing update from remote peer bc8713ea524e4655b599a1619c945cc4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:30.058462 25247 consensus_queue.cc:1035] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f70661a5b854a4b85823ad896d6ab12" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:30.058897 25229 consensus_queue.cc:1035] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4470d8736a04cd4b16b8ffda7aba8e5" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 41603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:30.061767 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.060958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.194:46471)
0812 11:24:30.061439 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.061569 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
I20260812 11:24:30.062427 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.061136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.195:60521)
0812 11:24:30.062063 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.062209 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":785}
I20260812 11:24:30.067822 25256 mvcc.cc:205] Tried to move back new op lower bound from 7317642731749597184 to 7317642731536257024. Current Snapshot: MvccSnapshot[applied={T|T < 7317642731749597184}]
I20260812 11:24:30.069968 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.069196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:51239)
0812 11:24:30.069677 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.069793 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
I20260812 11:24:30.071796 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.069807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.194:60187)
0812 11:24:30.070067 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.070153 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260812 11:24:30.075346 25237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.074720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.193:39047)
0812 11:24:30.075005 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.075114 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260812 11:24:30.077494 25254 mvcc.cc:205] Tried to move back new op lower bound from 7317642731749597184 to 7317642731536257024. Current Snapshot: MvccSnapshot[applied={T|T < 7317642731749597184}]
I20260812 11:24:30.079015 25227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.078282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.12.110.195:60013)
0812 11:24:30.078655 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.078777 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260812 11:24:30.090013 24313 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:30.090366 24313 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:30.096704 24313 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6dc45d4b35ca438a8a852f366e9aef2e: Sending DeleteTablet for 3 replicas of tablet a4193cc5f1e44ff38ad52e6fdb06d730
I20260812 11:24:30.097704 24720 tablet_service.cc:1514] Processing DeleteTablet for tablet a4193cc5f1e44ff38ad52e6fdb06d730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:30 UTC) from {username='oryx', principal='oryx/127.12.110.252@KRBTEST.COM'} at 127.0.0.1:57204
I20260812 11:24:30.098390 24502 tablet_service.cc:1514] Processing DeleteTablet for tablet a4193cc5f1e44ff38ad52e6fdb06d730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:30 UTC) from {username='oryx', principal='oryx/127.12.110.252@KRBTEST.COM'} at 127.0.0.1:39446
I20260812 11:24:30.098498 24937 tablet_service.cc:1514] Processing DeleteTablet for tablet a4193cc5f1e44ff38ad52e6fdb06d730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:30 UTC) from {username='oryx', principal='oryx/127.12.110.252@KRBTEST.COM'} at 127.0.0.1:40740
I20260812 11:24:30.099747 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 24279
I20260812 11:24:30.110132 25269 tablet_replica.cc:330] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: stopping tablet replica
I20260812 11:24:30.110416 25269 raft_consensus.cc:2229] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:30.110821 25269 raft_consensus.cc:2258] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:30.114485 25268 tablet_replica.cc:330] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: stopping tablet replica
I20260812 11:24:30.114717 25268 raft_consensus.cc:2229] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:30.115350 25267 tablet_replica.cc:330] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: stopping tablet replica
I20260812 11:24:30.115345 25268 raft_consensus.cc:2258] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:30.115571 25267 raft_consensus.cc:2229] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:30.115931 25267 raft_consensus.cc:2258] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:30.117082 25268 ts_tablet_manager.cc:1905] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:30.117153 25267 ts_tablet_manager.cc:1905] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:30.119130 25269 ts_tablet_manager.cc:1905] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:30.123757 25267 ts_tablet_manager.cc:1918] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:30.123900 25267 log.cc:1196] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/a4193cc5f1e44ff38ad52e6fdb06d730
I20260812 11:24:30.124421 25267 ts_tablet_manager.cc:1939] T a4193cc5f1e44ff38ad52e6fdb06d730 P b4470d8736a04cd4b16b8ffda7aba8e5: Deleting consensus metadata
I20260812 11:24:30.125646 24283 catalog_manager.cc:4841] TS b4470d8736a04cd4b16b8ffda7aba8e5 (127.12.110.194:41603): tablet a4193cc5f1e44ff38ad52e6fdb06d730 (table test-table [id=6c3c09c4555e4aa8a566c62a4462f273]) successfully deleted
I20260812 11:24:30.126124 25211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.101807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39462 (local address 127.12.110.193:46721)
0812 11:24:30.102016 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:30.102025 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:30.102049 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:24:30.102118 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:30.102126 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:30.102223 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:30.102348 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:30.104454 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.105939 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.108290 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.109194 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.109356 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:30.114047 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:30.114066 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:30.114087 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:30.114148 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:30.118405 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:30.120149 (+ 1744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:30.120158 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:30.120162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:30.120319 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:30.124379 (+ 4060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:30.124385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:30.124388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:30.124951 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:30.125298 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:30.125541 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 11:24:30.125784 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:24:30.126986 24602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:30.129115 25269 ts_tablet_manager.cc:1918] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:30.129271 25269 log.cc:1196] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/a4193cc5f1e44ff38ad52e6fdb06d730
I20260812 11:24:30.129817 25269 ts_tablet_manager.cc:1939] T a4193cc5f1e44ff38ad52e6fdb06d730 P 7f70661a5b854a4b85823ad896d6ab12: Deleting consensus metadata
I20260812 11:24:30.131052 24283 catalog_manager.cc:4841] TS 7f70661a5b854a4b85823ad896d6ab12 (127.12.110.195:45185): tablet a4193cc5f1e44ff38ad52e6fdb06d730 (table test-table [id=6c3c09c4555e4aa8a566c62a4462f273]) successfully deleted
I20260812 11:24:30.228511 25268 ts_tablet_manager.cc:1918] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:30.228956 25268 log.cc:1196] T a4193cc5f1e44ff38ad52e6fdb06d730 P bc8713ea524e4655b599a1619c945cc4: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/a4193cc5f1e44ff38ad52e6fdb06d730
I20260812 11:24:30.229477 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 24279
I20260812 11:24:30.258841 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 24624
I20260812 11:24:30.259161 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.258709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39466)
0812 11:24:30.258883 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.259007 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 11:24:30.259502 24283 catalog_manager.cc:4584] TS bc8713ea524e4655b599a1619c945cc4 (127.12.110.193:46721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4193cc5f1e44ff38ad52e6fdb06d730: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
I20260812 11:24:30.274233 25246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.259586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.12.110.194:41603)
0812 11:24:30.259774 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:30.259780 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:30.259897 (+ 117us) server_negotiation.cc:407] Connection header received
0812 11:24:30.260122 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:30.260128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:30.260215 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:30.260321 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:30.261741 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.262478 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.263832 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.264242 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.264376 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:30.267455 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:30.267471 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:30.267486 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:30.267521 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:30.270765 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:30.271536 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:30.271549 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:30.271555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:30.271754 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:30.272268 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:30.272275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:30.272280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:30.272799 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:30.273165 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:30.273507 (+ 342us) server_negotiation.cc:299] Negotiation successful
0812 11:24:30.273770 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:24:30.275111 24806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:30.318403 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.317914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38548)
0812 11:24:30.318107 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.318209 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 11:24:30.319443 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.319097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38564)
0812 11:24:30.319250 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.319331 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260812 11:24:30.320117 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.319831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39482)
0812 11:24:30.319939 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.320000 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260812 11:24:30.321024 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.320719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39498)
0812 11:24:30.320868 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.320928 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:24:30.389731 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.389251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39504)
0812 11:24:30.389458 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.389569 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 11:24:30.390766 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.390414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39506)
0812 11:24:30.390592 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.390663 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260812 11:24:30.403026 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 24624
I20260812 11:24:30.437734 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 24834
I20260812 11:24:30.454118 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.438623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.12.110.195:45185)
0812 11:24:30.438825 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:30.438831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:30.438852 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:24:30.438955 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:30.438965 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:30.439028 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:30.439144 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:30.440729 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.441860 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.443204 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:30.443613 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:30.443754 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:30.446922 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:30.446934 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:30.446951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:30.446996 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:30.450361 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:30.451927 (+ 1566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:30.451937 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:30.451939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:30.452091 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:30.452504 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:30.452508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:30.452510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:30.452934 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:30.453265 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:30.453553 (+ 288us) server_negotiation.cc:299] Negotiation successful
0812 11:24:30.453813 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:24:30.454793 25045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:30.493350 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.492442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39518)
0812 11:24:30.492852 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.493005 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260812 11:24:30.494719 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.494239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39528)
0812 11:24:30.494422 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.494518 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260812 11:24:30.557142 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 24834
I20260812 11:24:30.586091 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 23786
I20260812 11:24:30.587651 24052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:30.666728 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.666124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39534)
0812 11:24:30.666369 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.666471 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 11:24:30.667771 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.667421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:46721 (local address 127.0.0.1:39542)
0812 11:24:30.667596 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.667664 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:46721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 11:24:30.713972 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 23786
I20260812 11:24:30.720672 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.720234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38568)
0812 11:24:30.720409 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.720507 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 11:24:30.721633 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:30.721316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:44299 (local address 127.0.0.1:38584)
0812 11:24:30.721474 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:30.721537 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:44299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260812 11:24:30.742642 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 23942
I20260812 11:24:30.743883 24357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:30.868722 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 23942
2026-08-12T11:24:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10741 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:24:30 dist-test-slave-h6n0 krb5kdc[25286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:24:30 dist-test-slave-h6n0 krb5kdc[25286](info): set up 2 sockets
Aug 12 11:24:30 dist-test-slave-h6n0 krb5kdc[25286](info): commencing operation
krb5kdc: starting...
W20260812 11:24:32.259608 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:24:32 dist-test-slave-h6n0 krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:32Z Disabled control of system clock
WARNING: no policy specified for oryx/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab.
Entry for principal oryx/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab.
I20260812 11:24:32.552302 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37329
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab
--principal=oryx/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:32.916618 25306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:32.916908 25306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:32.917078 25306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:32.924146 25306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:32.924245 25306 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260812 11:24:32.924342 25306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:32.924449 25306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:32.924531 25306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:32.924646 25306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:32.934342 25306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.254.keytab
--principal=oryx/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:32.935671 25306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:32.937054 25306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:32.949129 25317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:32.949159 25322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:32.949589 25316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:32.950037 25306 server_base.cc:1034] running on GCE node
I20260812 11:24:32.950944 25306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:32.952386 25306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:32.953613 25306 hybrid_clock.cc:648] HybridClock initialized: now 1786533872953548 us; error 31 us; skew 500 ppm
I20260812 11:24:32.953953 25306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:32 dist-test-slave-h6n0 krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533872, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:32.957309 25306 init.cc:377] Logged in from keytab as oryx/127.12.110.254@KRBTEST.COM (short username oryx)
I20260812 11:24:32.959026 25306 webserver.cc:460] Webserver started at http://127.12.110.254:33347/ using document root <none> and password file <none>
I20260812 11:24:32.959733 25306 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:32.959821 25306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:32.960160 25306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:32.963420 25306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "b25772c68b764fe4babb66b80064dd67"
format_stamp: "Formatted at 2026-08-12 11:24:32 on dist-test-slave-h6n0"
server_key: "79e8dcf0dda1679ce6067f6065216d13"
server_key_iv: "d39726bf20e2df748c04b0967dc22a25"
server_key_version: "encryptionkey@0"
I20260812 11:24:32.964236 25306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "b25772c68b764fe4babb66b80064dd67"
format_stamp: "Formatted at 2026-08-12 11:24:32 on dist-test-slave-h6n0"
server_key: "79e8dcf0dda1679ce6067f6065216d13"
server_key_iv: "d39726bf20e2df748c04b0967dc22a25"
server_key_version: "encryptionkey@0"
I20260812 11:24:32.969304 25306 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 11:24:32.972774 25339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:32.974681 25306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 11:24:32.974864 25306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "b25772c68b764fe4babb66b80064dd67"
format_stamp: "Formatted at 2026-08-12 11:24:32 on dist-test-slave-h6n0"
server_key: "79e8dcf0dda1679ce6067f6065216d13"
server_key_iv: "d39726bf20e2df748c04b0967dc22a25"
server_key_version: "encryptionkey@0"
I20260812 11:24:32.975096 25306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:33.001964 25306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:33.006108 25306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:33.006399 25306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:33.030495 25306 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37329
I20260812 11:24:33.030519 25432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37329 every 8 connection(s)
I20260812 11:24:33.031919 25306 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:33.033810 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 25306
I20260812 11:24:33.034049 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:33.034960 12731 external_mini_cluster.cc:1428] Setting key 53c2f6daf78b4db6cc2c554a4f0b4739
I20260812 11:24:33.036733 25433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:33.048647 25433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Bootstrap starting.
Aug 12 11:24:33 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:33.052188 25433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:33.053434 25433 log.cc:824] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:33.056305 25433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: No bootstrap required, opened a new log
I20260812 11:24:33.058272 25437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:33.037348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.12.110.254:37329)
0812 11:24:33.038233 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:33.038245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:33.038286 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:24:33.040042 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:33.040063 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:33.040439 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:33.040872 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:33.042497 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.043969 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.045270 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.046011 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.046283 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:33.050936 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:33.050944 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:33.050957 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:33.050993 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:33.054214 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:33.055031 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:33.055040 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:33.055047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:33.055209 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:33.055617 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:33.055622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:33.055625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:33.056128 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:33.056448 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:33.056905 (+ 457us) server_negotiation.cc:299] Negotiation successful
0812 11:24:33.057315 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":467,"threads_started":1}
I20260812 11:24:33.065573 25433 raft_consensus.cc:348] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b25772c68b764fe4babb66b80064dd67" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37329 } }
I20260812 11:24:33.066254 25433 raft_consensus.cc:374] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:33.066434 25433 raft_consensus.cc:729] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25772c68b764fe4babb66b80064dd67, State: Initialized, Role: FOLLOWER
I20260812 11:24:33.067066 25433 consensus_queue.cc:260] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b25772c68b764fe4babb66b80064dd67" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37329 } }
I20260812 11:24:33.067363 25433 raft_consensus.cc:388] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:33.067525 25433 raft_consensus.cc:482] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:33.067692 25433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:33.069810 25433 raft_consensus.cc:504] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b25772c68b764fe4babb66b80064dd67" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37329 } }
I20260812 11:24:33.070262 25433 leader_election.cc:302] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b25772c68b764fe4babb66b80064dd67; no voters:
I20260812 11:24:33.070866 25433 leader_election.cc:288] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:33.071024 25439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:33.071779 25439 raft_consensus.cc:686] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [term 1 LEADER]: Becoming Leader. State: Replica: b25772c68b764fe4babb66b80064dd67, State: Running, Role: LEADER
I20260812 11:24:33.072346 25439 consensus_queue.cc:237] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b25772c68b764fe4babb66b80064dd67" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37329 } }
I20260812 11:24:33.073092 25433 sys_catalog.cc:564] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:33.076150 25442 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b25772c68b764fe4babb66b80064dd67. Latest consensus state: current_term: 1 leader_uuid: "b25772c68b764fe4babb66b80064dd67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b25772c68b764fe4babb66b80064dd67" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37329 } } }
I20260812 11:24:33.076063 25441 sys_catalog.cc:455] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b25772c68b764fe4babb66b80064dd67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b25772c68b764fe4babb66b80064dd67" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37329 } } }
I20260812 11:24:33.076715 25442 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:33.076812 25441 sys_catalog.cc:458] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:33.080282 25451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:33.087453 25451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:33.098300 25451 catalog_manager.cc:1347] Generated new cluster ID: fe79e121c9f84a60832a3d012380fa73
I20260812 11:24:33.098459 25451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:33.118799 25451 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:33.120517 25451 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:33.133778 25451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Generated new TSK 0
I20260812 11:24:33.134819 25451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab.
Entry for principal oryx/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab.
I20260812 11:24:33.209770 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37329
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab
--principal=oryx/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:33.553052 25471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:33.553364 25471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:33.553570 25471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:33.561448 25471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:33.561564 25471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:33.561769 25471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:24:33.561875 25471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:33.572412 25471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab
--principal=oryx/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:33.573762 25471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:33.575129 25471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:33.588543 25486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:33.588828 25484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:33.588523 25489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:33.589321 25471 server_base.cc:1034] running on GCE node
I20260812 11:24:33.589942 25471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:33.590883 25471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:33.592093 25471 hybrid_clock.cc:648] HybridClock initialized: now 1786533873592040 us; error 27 us; skew 500 ppm
I20260812 11:24:33.592376 25471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:33 dist-test-slave-h6n0 krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533873, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:33.599040 25471 init.cc:377] Logged in from keytab as oryx/127.12.110.193@KRBTEST.COM (short username oryx)
I20260812 11:24:33.600421 25471 webserver.cc:460] Webserver started at http://127.12.110.193:34435/ using document root <none> and password file <none>
I20260812 11:24:33.601071 25471 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:33.601157 25471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:33.601415 25471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:33.604272 25471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "94f63c7bf490499c8537184bf309762f"
format_stamp: "Formatted at 2026-08-12 11:24:33 on dist-test-slave-h6n0"
server_key: "77d4176c7f7ada008955781e7d130a29"
server_key_iv: "ba13f687466be77f4f3e1c900e724f58"
server_key_version: "encryptionkey@0"
I20260812 11:24:33.604992 25471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "94f63c7bf490499c8537184bf309762f"
format_stamp: "Formatted at 2026-08-12 11:24:33 on dist-test-slave-h6n0"
server_key: "77d4176c7f7ada008955781e7d130a29"
server_key_iv: "ba13f687466be77f4f3e1c900e724f58"
server_key_version: "encryptionkey@0"
I20260812 11:24:33.610235 25471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:24:33.613529 25508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:33.614840 25471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 11:24:33.615029 25471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "94f63c7bf490499c8537184bf309762f"
format_stamp: "Formatted at 2026-08-12 11:24:33 on dist-test-slave-h6n0"
server_key: "77d4176c7f7ada008955781e7d130a29"
server_key_iv: "ba13f687466be77f4f3e1c900e724f58"
server_key_version: "encryptionkey@0"
I20260812 11:24:33.615267 25471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:33.642699 25471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:33.646919 25471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:33.647199 25471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:33.649852 25471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:33.649963 25471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:33.650127 25471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:33.650206 25471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:24:33 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533873, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:33.674623 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:33.650646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:33397 (local address 127.12.110.254:37329)
0812 11:24:33.651635 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:33.651642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:33.652682 (+ 1040us) server_negotiation.cc:407] Connection header received
0812 11:24:33.654811 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:33.654821 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:33.654929 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:33.655074 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:33.657368 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.658470 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.661142 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.661583 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.661721 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:33.666483 (+ 4762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:33.666488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:33.666491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:33.666522 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:33.669712 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:33.670429 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:33.670434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:33.670440 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:33.670629 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:33.671307 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:33.671315 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:33.671322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:33.671747 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:33.671912 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:33.674015 (+ 2103us) server_negotiation.cc:299] Negotiation successful
0812 11:24:33.674237 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":751,"threads_started":1}
I20260812 11:24:33.675562 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:33.651486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37329 (local address 127.12.110.193:33397)
0812 11:24:33.652498 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:33.652547 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:33.654470 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:33.655351 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:33.655361 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:33.656148 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:33.657120 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:33.657154 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.658656 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.658664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:33.660972 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:33.660985 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.661855 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.661864 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:33.662071 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:33.662863 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:33.662900 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:33.666264 (+ 3364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:33.669862 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:33.669871 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:33.669886 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:33.670298 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:33.670787 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:33.670794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:33.670799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:33.671106 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:33.672011 (+ 905us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:33.672022 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:33.672611 (+ 589us) client_negotiation.cc:769] Sending connection context
0812 11:24:33.672985 (+ 374us) client_negotiation.cc:240] Negotiation successful
0812 11:24:33.673673 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":488,"threads_started":1}
I20260812 11:24:33.696121 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:33.682172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37329 (local address 127.12.110.193:55565)
0812 11:24:33.682983 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:33.683008 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:33.683195 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:33.683588 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:33.683592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:33.684124 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:33.684463 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:33.684474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.685438 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.685444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:33.686482 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:33.686494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.687560 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.687571 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:33.687743 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:33.688471 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:33.688488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:33.689726 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:33.693101 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:33.693109 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:33.693116 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:33.693842 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:33.694274 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:33.694279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:33.694283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:33.694508 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:33.695242 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:33.695250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:33.695451 (+ 201us) client_negotiation.cc:769] Sending connection context
0812 11:24:33.695573 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 11:24:33.695802 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":393,"threads_started":1}
I20260812 11:24:33.696575 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:33.682109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:55565 (local address 127.12.110.254:37329)
0812 11:24:33.682273 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:33.682277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:33.683102 (+ 825us) server_negotiation.cc:407] Connection header received
0812 11:24:33.683367 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:33.683375 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:33.683458 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:33.683563 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:33.684598 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.685294 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.686719 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:33.687350 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:33.687492 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:33.689960 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:33.689964 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:33.689967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:33.689997 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:33.692971 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:33.693988 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:33.693993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:33.693996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:33.694153 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:33.694656 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:33.694663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:33.694669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:33.695095 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:33.695266 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:33.695756 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 11:24:33.696063 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260812 11:24:33.702375 25471 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:40663
I20260812 11:24:33.702399 25704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:40663 every 8 connection(s)
I20260812 11:24:33.703545 25471 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:33.705736 25706 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37329
I20260812 11:24:33.705986 25706 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:33.706661 25706 heartbeater.cc:511] Master 127.12.110.254:37329 requested a full tablet report, sending...
I20260812 11:24:33.708622 25359 ts_manager.cc:194] Registered new tserver with Master: 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663)
I20260812 11:24:33.709292 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 25471
I20260812 11:24:33.709496 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:24:33.710152 12731 external_mini_cluster.cc:1428] Setting key 5dfe3d465550f02aa37f523457392003
I20260812 11:24:33.710142 25359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:33397
WARNING: no policy specified for oryx/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab.
Entry for principal oryx/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab.
I20260812 11:24:33.769966 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37329
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab
--principal=oryx/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:34.103570 25710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:34.103982 25710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:34.104184 25710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:34.111593 25710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:34.111743 25710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:34.111908 25710 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:24:34.112061 25710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:34.121824 25710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab
--principal=oryx/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:34.123044 25710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:34.124409 25710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:34.137392 25718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:34.137359 25719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:34.137355 25722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:34.140002 25710 server_base.cc:1034] running on GCE node
I20260812 11:24:34.140674 25710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:34.141579 25710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:34.142850 25710 hybrid_clock.cc:648] HybridClock initialized: now 1786533874142758 us; error 59 us; skew 500 ppm
I20260812 11:24:34.143146 25710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533874, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:34.146746 25710 init.cc:377] Logged in from keytab as oryx/127.12.110.194@KRBTEST.COM (short username oryx)
I20260812 11:24:34.148283 25710 webserver.cc:460] Webserver started at http://127.12.110.194:42097/ using document root <none> and password file <none>
I20260812 11:24:34.148983 25710 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:34.149117 25710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:34.149443 25710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:34.152369 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "538486cabc2d4e67a4285beb67ff43f9"
format_stamp: "Formatted at 2026-08-12 11:24:34 on dist-test-slave-h6n0"
server_key: "e1d166cec3b51a67da407c3b87c74de2"
server_key_iv: "028d21a3c44ee2e437fdbe78c5fa2334"
server_key_version: "encryptionkey@0"
I20260812 11:24:34.153174 25710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "538486cabc2d4e67a4285beb67ff43f9"
format_stamp: "Formatted at 2026-08-12 11:24:34 on dist-test-slave-h6n0"
server_key: "e1d166cec3b51a67da407c3b87c74de2"
server_key_iv: "028d21a3c44ee2e437fdbe78c5fa2334"
server_key_version: "encryptionkey@0"
I20260812 11:24:34.158726 25710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260812 11:24:34.162645 25735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:34.164548 25710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:24:34.164736 25710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "538486cabc2d4e67a4285beb67ff43f9"
format_stamp: "Formatted at 2026-08-12 11:24:34 on dist-test-slave-h6n0"
server_key: "e1d166cec3b51a67da407c3b87c74de2"
server_key_iv: "028d21a3c44ee2e437fdbe78c5fa2334"
server_key_version: "encryptionkey@0"
I20260812 11:24:34.164968 25710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:34.182950 25710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:34.187659 25710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:34.187947 25710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:34.191035 25710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:34.191174 25710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:34.191355 25710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:34.191429 25710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533874, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:34.218327 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.191399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:33005 (local address 127.12.110.254:37329)
0812 11:24:34.191590 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.191601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.193056 (+ 1455us) server_negotiation.cc:407] Connection header received
0812 11:24:34.195144 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.195150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.195250 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.195367 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:34.197529 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.198859 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.201475 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.202149 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.202299 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.208306 (+ 6007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:34.208314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:34.208321 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:34.208369 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:34.212659 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.214458 (+ 1799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.214464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.214468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.214695 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.215596 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.215603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.215608 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.216087 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:34.216271 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.217562 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.217859 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:24:34.220813 25748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.192040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37329 (local address 127.12.110.194:33005)
0812 11:24:34.192888 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:34.192925 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:34.194873 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:34.195650 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:34.195667 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:34.196446 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:34.197282 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.197313 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.199054 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.199061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.201274 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.201291 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.202413 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.202420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.202780 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.203608 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:34.203631 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:34.208006 (+ 4375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:34.212929 (+ 4923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.212954 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.212987 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.214242 (+ 1255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.214871 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.214879 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.214886 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.215406 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.216331 (+ 925us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:34.216346 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:34.216986 (+ 640us) client_negotiation.cc:769] Sending connection context
0812 11:24:34.217429 (+ 443us) client_negotiation.cc:240] Negotiation successful
0812 11:24:34.218209 (+ 780us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":376,"threads_started":1}
I20260812 11:24:34.242331 25710 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:42249
I20260812 11:24:34.242349 25906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:42249 every 8 connection(s)
I20260812 11:24:34.244270 25710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:34.246953 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 25710
I20260812 11:24:34.247138 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:24:34.247589 12731 external_mini_cluster.cc:1428] Setting key cbfb4ce4e99f304df06a5611aded67c8
I20260812 11:24:34.248347 25908 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37329
I20260812 11:24:34.248596 25908 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:34.249475 25908 heartbeater.cc:511] Master 127.12.110.254:37329 requested a full tablet report, sending...
I20260812 11:24:34.251194 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.233466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:40279 (local address 127.12.110.254:37329)
0812 11:24:34.233706 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.233714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.234576 (+ 862us) server_negotiation.cc:407] Connection header received
0812 11:24:34.234988 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.235000 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.235263 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.235422 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:34.237239 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.238209 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.240977 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.241655 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.241922 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.244560 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:34.244567 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:34.244574 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:34.244625 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:34.247989 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.248835 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.248843 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.248847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.249000 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.249489 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.249493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.249495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.249979 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:34.250137 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.250627 (+ 490us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.250843 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:24:34.251277 25875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.233507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37329 (local address 127.12.110.194:40279)
0812 11:24:34.234409 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:34.234432 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:34.234650 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:34.235656 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:34.235661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:34.236557 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:34.237076 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.237089 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.238427 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.238434 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.240775 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.240792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.242010 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.242024 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.242264 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.243149 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:34.243173 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:34.244324 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:34.248136 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.248144 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.248147 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.248707 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.249123 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.249128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.249131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.249263 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.250128 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:34.250133 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:34.250331 (+ 198us) client_negotiation.cc:769] Sending connection context
0812 11:24:34.250486 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 11:24:34.250840 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":500,"threads_started":1}
I20260812 11:24:34.251418 25359 ts_manager.cc:194] Registered new tserver with Master: 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249)
I20260812 11:24:34.252713 25359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:33005
WARNING: no policy specified for oryx/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab.
Entry for principal oryx/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab.
I20260812 11:24:34.311913 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37329
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab
--principal=oryx/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33629
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:34.666659 25921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:34.666922 25921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:34.667024 25921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:34.674481 25921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:34.674592 25921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:34.674674 25921 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260812 11:24:34.674754 25921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:34.684677 25921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33629
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab
--principal=oryx/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.12.110.254:37329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:34.685846 25921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:34.687521 25921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:34.700793 25934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:34.700855 25931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:34.701210 25928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:34.704654 25921 server_base.cc:1034] running on GCE node
I20260812 11:24:34.705268 25921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:34.706265 25921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:34.707551 25921 hybrid_clock.cc:648] HybridClock initialized: now 1786533874707468 us; error 49 us; skew 500 ppm
I20260812 11:24:34.707873 25921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533874, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:34.711364 25921 init.cc:377] Logged in from keytab as oryx/127.12.110.195@KRBTEST.COM (short username oryx)
I20260812 11:24:34.712879 25921 webserver.cc:460] Webserver started at http://127.12.110.195:36891/ using document root <none> and password file <none>
I20260812 11:24:34.713214 25706 heartbeater.cc:503] Master 127.12.110.254:37329 was elected leader, sending a full tablet report...
I20260812 11:24:34.713743 25921 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:34.713868 25921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:34.714311 25921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:34.717517 25921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "69ca1916207d48e3b21045e133391a3f"
format_stamp: "Formatted at 2026-08-12 11:24:34 on dist-test-slave-h6n0"
server_key: "ba95d1bd50e06419a120a599c0526775"
server_key_iv: "f8b02625d9b0ed7dd881b7f2e1d774bd"
server_key_version: "encryptionkey@0"
I20260812 11:24:34.718263 25921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "69ca1916207d48e3b21045e133391a3f"
format_stamp: "Formatted at 2026-08-12 11:24:34 on dist-test-slave-h6n0"
server_key: "ba95d1bd50e06419a120a599c0526775"
server_key_iv: "f8b02625d9b0ed7dd881b7f2e1d774bd"
server_key_version: "encryptionkey@0"
I20260812 11:24:34.724215 25921 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260812 11:24:34.727628 25943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:34.729001 25921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:24:34.729151 25921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "69ca1916207d48e3b21045e133391a3f"
format_stamp: "Formatted at 2026-08-12 11:24:34 on dist-test-slave-h6n0"
server_key: "ba95d1bd50e06419a120a599c0526775"
server_key_iv: "f8b02625d9b0ed7dd881b7f2e1d774bd"
server_key_version: "encryptionkey@0"
I20260812 11:24:34.729355 25921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:34.762735 25921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:34.768646 25921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:34.768949 25921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:34.772073 25921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:34.772207 25921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:34.772372 25921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:34.772509 25921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533874, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:34.798849 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.772562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:56025 (local address 127.12.110.254:37329)
0812 11:24:34.773137 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.773145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.774331 (+ 1186us) server_negotiation.cc:407] Connection header received
0812 11:24:34.776476 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.776481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.776573 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.776695 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:34.779300 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.780407 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.784285 (+ 3878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.784754 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.784922 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.790814 (+ 5892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:34.790824 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:34.790832 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:34.790896 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:34.795072 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.796290 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.796294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.796297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.796438 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.796940 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.796947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.796955 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.797238 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:34.797421 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.798182 (+ 761us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.798412 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":296,"threads_started":1}
I20260812 11:24:34.800477 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.773306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37329 (local address 127.12.110.195:56025)
0812 11:24:34.774123 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:34.774204 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:34.776194 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:34.776941 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:34.776949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:34.777804 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:34.778938 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.778972 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.780588 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.780596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.783723 (+ 3127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.783854 (+ 131us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.785050 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.785061 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.785278 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.786241 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:34.786275 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:34.790434 (+ 4159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:34.795387 (+ 4953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.795399 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.795415 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.796093 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.796564 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.796572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.796577 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.796804 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.797377 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:34.797388 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:34.797788 (+ 400us) client_negotiation.cc:769] Sending connection context
0812 11:24:34.798070 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 11:24:34.798637 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":356,"threads_started":1}
I20260812 11:24:34.828599 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.812008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37329 (local address 127.12.110.195:50467)
0812 11:24:34.812647 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:34.812666 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:34.812863 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:34.813226 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:34.813231 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:34.813726 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:34.814016 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.814026 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.815169 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.815176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.817427 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.817448 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.818234 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.818243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.818489 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.819312 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:34.819343 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:34.820281 (+ 938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:34.824738 (+ 4457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.824746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.824752 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.825550 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.825939 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:34.825947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:34.825953 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:34.826151 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:34.826723 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:34.826731 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:34.827004 (+ 273us) client_negotiation.cc:769] Sending connection context
0812 11:24:34.827199 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 11:24:34.827579 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":377,"threads_started":1}
I20260812 11:24:34.829341 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.811589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:50467 (local address 127.12.110.254:37329)
0812 11:24:34.811779 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.811783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.812769 (+ 986us) server_negotiation.cc:407] Connection header received
0812 11:24:34.812997 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.813004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.813083 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.813192 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:34.814162 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.814994 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.817594 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.818027 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.818122 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.820512 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:34.820521 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:34.820529 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:34.820580 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:34.824601 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.825680 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.825684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.825686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.825817 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.826266 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.826270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.826272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.826599 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:34.826761 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.828925 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.829095 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:24:34.837317 25921 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:41305
I20260812 11:24:34.837402 26155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:41305 every 8 connection(s)
I20260812 11:24:34.838490 25921 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:34.840484 26156 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37329
I20260812 11:24:34.840674 26156 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:34.841233 26156 heartbeater.cc:511] Master 127.12.110.254:37329 requested a full tablet report, sending...
I20260812 11:24:34.842571 25359 ts_manager.cc:194] Registered new tserver with Master: 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305)
I20260812 11:24:34.843412 25359 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:56025
I20260812 11:24:34.846635 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 25921
I20260812 11:24:34.846812 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:24:34.847290 12731 external_mini_cluster.cc:1428] Setting key 90bffb977aca4e338b0a8fb3ea784d5f
I20260812 11:24:34.850219 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:24:34.918286 25956 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:34.906869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.12.110.254:37329)
0812 11:24:34.907112 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.907118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.907244 (+ 126us) server_negotiation.cc:407] Connection header received
0812 11:24:34.907419 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.907422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.907487 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.907576 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:34.909202 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.910205 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.911750 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.912188 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.912292 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.918061 (+ 5769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57798: BlockingRecv error: recv got EOF from 127.0.0.1:57798 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 11:24:34 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:34.935434 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.921617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.12.110.254:37329)
0812 11:24:34.921845 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.921851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.922120 (+ 269us) server_negotiation.cc:407] Connection header received
0812 11:24:34.922314 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.922318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.922395 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.922504 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:34.923718 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.924446 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.925678 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.926091 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.926226 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.929984 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:34.929990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:34.929993 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:34.930025 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:34.932667 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.933436 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.933439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.933442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.933581 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:34.933978 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:34.933983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:34.933986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:34.934259 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:34.934588 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.934899 (+ 311us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.935143 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:24:34.939918 25359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:34.947090 25359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:34.983904 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.962195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.12.110.194:42249)
0812 11:24:34.975990 (+ 13795us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.975998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.976028 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:24:34.976123 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.976131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.976356 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.976795 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:34.977564 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.979227 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.981373 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.982705 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.982899 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.983087 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.983408 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.983582 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13541,"thread_start_us":13394,"threads_started":1}
I20260812 11:24:34.984130 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.961907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:42249 (local address 127.0.0.1:35668)
0812 11:24:34.973160 (+ 11253us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:34.973205 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:34.973424 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:34.976568 (+ 3144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:34.976572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:34.976601 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:34.977300 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.977314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.979420 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.979427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.981212 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.981225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.982953 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.982960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.983159 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.983216 (+ 57us) client_negotiation.cc:769] Sending connection context
0812 11:24:34.983348 (+ 132us) client_negotiation.cc:240] Negotiation successful
0812 11:24:34.983606 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10802,"mutex_wait_us":96,"thread_start_us":11965,"threads_started":1}
I20260812 11:24:34.985092 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.962914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.12.110.193:40663)
0812 11:24:34.971560 (+ 8646us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.971575 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.974584 (+ 3009us) server_negotiation.cc:407] Connection header received
0812 11:24:34.974706 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.974715 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.976011 (+ 1296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.976218 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:34.977841 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.979398 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.981947 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.983224 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.983413 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.983567 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.984298 (+ 731us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.984782 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8398,"thread_start_us":8998,"threads_started":1}
I20260812 11:24:34.985737 26191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.962245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:40663 (local address 127.0.0.1:43696)
0812 11:24:34.974247 (+ 12002us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:34.974269 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:34.974412 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:34.976534 (+ 2122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:34.976545 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:34.976635 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:34.977641 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.977662 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.979590 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.979596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.981783 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.981799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.983719 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.983725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.983940 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.983985 (+ 45us) client_negotiation.cc:769] Sending connection context
0812 11:24:34.984084 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:24:34.985369 (+ 1285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11842,"mutex_wait_us":182,"spinlock_wait_cycles":14720,"thread_start_us":13051,"threads_started":1}
I20260812 11:24:34.986905 26196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.962839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41305 (local address 127.0.0.1:55490)
0812 11:24:34.975648 (+ 12809us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:34.975668 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:34.975797 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:34.976537 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:34.976551 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:34.976678 (+ 127us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:34.977679 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.977695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.981856 (+ 4161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.981862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.983991 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:34.984007 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.986045 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.986051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:34.986219 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.986252 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:24:34.986341 (+ 89us) client_negotiation.cc:240] Negotiation successful
0812 11:24:34.986465 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12622,"mutex_wait_us":168,"spinlock_wait_cycles":159872,"thread_start_us":9699,"threads_started":1}
I20260812 11:24:34.987701 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:34.963547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.12.110.195:41305)
0812 11:24:34.975026 (+ 11479us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:34.975052 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:34.975858 (+ 806us) server_negotiation.cc:407] Connection header received
0812 11:24:34.975959 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:34.975968 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:34.976238 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:34.976416 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:34.978793 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.981696 (+ 2903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.984197 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:34.985530 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:34.985737 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:34.985906 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:34.987189 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0812 11:24:34.987352 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11181,"thread_start_us":10971,"threads_started":1}
I20260812 11:24:34.988694 25800 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae477a106d9488484b444b9c62a4dbd (DEFAULT_TABLE table=test-table [id=d1ff65e86e4c439bb2d5e07b562a8c76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:34.988914 25591 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae477a106d9488484b444b9c62a4dbd (DEFAULT_TABLE table=test-table [id=d1ff65e86e4c439bb2d5e07b562a8c76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:34.990063 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae477a106d9488484b444b9c62a4dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:34.990296 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae477a106d9488484b444b9c62a4dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:34.991289 26038 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae477a106d9488484b444b9c62a4dbd (DEFAULT_TABLE table=test-table [id=d1ff65e86e4c439bb2d5e07b562a8c76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:34.992571 26038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae477a106d9488484b444b9c62a4dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:35.001192 26205 tablet_bootstrap.cc:492] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Bootstrap starting.
I20260812 11:24:35.004642 26206 tablet_bootstrap.cc:492] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Bootstrap starting.
I20260812 11:24:35.005714 26205 tablet_bootstrap.cc:654] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.006729 26208 tablet_bootstrap.cc:492] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Bootstrap starting.
I20260812 11:24:35.007062 26205 log.cc:824] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:35.008121 26206 tablet_bootstrap.cc:654] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.009394 26206 log.cc:824] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:35.009771 26208 tablet_bootstrap.cc:654] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.010294 26205 tablet_bootstrap.cc:492] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: No bootstrap required, opened a new log
I20260812 11:24:35.010630 26205 ts_tablet_manager.cc:1397] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260812 11:24:35.011024 26208 log.cc:824] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:35.012641 26206 tablet_bootstrap.cc:492] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: No bootstrap required, opened a new log
I20260812 11:24:35.012938 26206 ts_tablet_manager.cc:1397] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 11:24:35.014220 26208 tablet_bootstrap.cc:492] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: No bootstrap required, opened a new log
I20260812 11:24:35.014492 26208 ts_tablet_manager.cc:1397] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 11:24:35.019217 26205 raft_consensus.cc:348] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.019744 26205 raft_consensus.cc:374] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.019848 26205 raft_consensus.cc:729] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538486cabc2d4e67a4285beb67ff43f9, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.020504 26205 consensus_queue.cc:260] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.021631 25908 heartbeater.cc:503] Master 127.12.110.254:37329 was elected leader, sending a full tablet report...
I20260812 11:24:35.022147 26205 ts_tablet_manager.cc:1428] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260812 11:24:35.023306 26206 raft_consensus.cc:348] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.023610 26206 raft_consensus.cc:374] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.023671 26206 raft_consensus.cc:729] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f63c7bf490499c8537184bf309762f, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.024271 26206 consensus_queue.cc:260] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.025825 26208 raft_consensus.cc:348] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.026186 26206 ts_tablet_manager.cc:1428] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260812 11:24:35.026242 26208 raft_consensus.cc:374] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.026347 26208 raft_consensus.cc:729] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca1916207d48e3b21045e133391a3f, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.027041 26208 consensus_queue.cc:260] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.028239 26156 heartbeater.cc:503] Master 127.12.110.254:37329 was elected leader, sending a full tablet report...
I20260812 11:24:35.028751 26208 ts_tablet_manager.cc:1428] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
W20260812 11:24:35.090158 26158 tablet.cc:2369] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:24:35.206259 25707 tablet.cc:2369] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:35.217644 26215 raft_consensus.cc:482] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:35.218012 26215 raft_consensus.cc:504] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.220093 26215 leader_election.cc:288] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249), 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663)
I20260812 11:24:35.226022 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.220466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:57947 (local address 127.12.110.194:42249)
0812 11:24:35.220683 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.220689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.220934 (+ 245us) server_negotiation.cc:407] Connection header received
0812 11:24:35.221131 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.221135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.221196 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.221317 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:35.221793 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.222582 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.224232 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.225119 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.225287 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.225361 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.225645 (+ 284us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.225770 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:24:35.226022 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.220568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:42249 (local address 127.12.110.195:57947)
0812 11:24:35.220804 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.220825 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.220973 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.221340 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.221345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.221385 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:35.221632 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.221641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.222715 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.222722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.224066 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.224078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.225267 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.225273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.225478 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.225502 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.225618 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.225748 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":1}
I20260812 11:24:35.226897 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.220466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:45527 (local address 127.12.110.193:40663)
0812 11:24:35.220685 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.220691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.221720 (+ 1029us) server_negotiation.cc:407] Connection header received
0812 11:24:35.221921 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.221925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.221998 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.222098 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:35.222627 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.223314 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.225146 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.225981 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.226158 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.226231 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.226547 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.226670 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:24:35.227205 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.220893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:40663 (local address 127.12.110.195:45527)
0812 11:24:35.221613 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.221627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.221765 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.222113 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.222117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.222140 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:35.222446 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.222454 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.223452 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.223458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.224944 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.224982 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.226200 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.226206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.226362 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.226384 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.226491 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.226632 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":392,"threads_started":1}
I20260812 11:24:35.227326 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae477a106d9488484b444b9c62a4dbd" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9" is_pre_election: true
I20260812 11:24:35.227835 25830 raft_consensus.cc:2454] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 0.
I20260812 11:24:35.228195 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae477a106d9488484b444b9c62a4dbd" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f63c7bf490499c8537184bf309762f" is_pre_election: true
I20260812 11:24:35.228574 25946 leader_election.cc:302] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538486cabc2d4e67a4285beb67ff43f9, 69ca1916207d48e3b21045e133391a3f; no voters:
I20260812 11:24:35.228749 25620 raft_consensus.cc:2454] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 0.
I20260812 11:24:35.229045 26215 raft_consensus.cc:2781] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:35.229209 26215 raft_consensus.cc:482] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:35.229336 26215 raft_consensus.cc:3037] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.231654 26215 raft_consensus.cc:504] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.232304 26215 leader_election.cc:288] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 election: Requested vote from peers 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249), 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663)
I20260812 11:24:35.232788 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae477a106d9488484b444b9c62a4dbd" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9"
I20260812 11:24:35.232841 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae477a106d9488484b444b9c62a4dbd" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f63c7bf490499c8537184bf309762f"
I20260812 11:24:35.233024 25830 raft_consensus.cc:3037] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.233039 25620 raft_consensus.cc:3037] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.235543 25620 raft_consensus.cc:2454] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 1.
I20260812 11:24:35.236120 25945 leader_election.cc:302] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca1916207d48e3b21045e133391a3f, 94f63c7bf490499c8537184bf309762f; no voters:
I20260812 11:24:35.236701 25830 raft_consensus.cc:2454] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 1.
I20260812 11:24:35.236717 26215 raft_consensus.cc:2781] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:35.237562 26215 raft_consensus.cc:686] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 1 LEADER]: Becoming Leader. State: Replica: 69ca1916207d48e3b21045e133391a3f, State: Running, Role: LEADER
I20260812 11:24:35.238194 26215 consensus_queue.cc:237] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.244112 25358 catalog_manager.cc:5495] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ca1916207d48e3b21045e133391a3f (127.12.110.195). New cstate: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } health_report { overall_health: HEALTHY } } }
W20260812 11:24:35.246912 25910 tablet.cc:2369] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:24:35.270758 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.265466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.12.110.195:41305)
0812 11:24:35.265699 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.265705 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.265725 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:24:35.265866 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.265870 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.265948 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.266103 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:35.266478 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.267249 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.268681 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.269074 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.269228 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.269631 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.270136 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.270295 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.270390 (+ 95us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.270471 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260812 11:24:35.278905 25830 raft_consensus.cc:1261] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Refusing update from remote peer 69ca1916207d48e3b21045e133391a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:35.278914 25620 raft_consensus.cc:1261] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Refusing update from remote peer 69ca1916207d48e3b21045e133391a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:35.279920 26220 consensus_queue.cc:1035] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:35.280282 26215 consensus_queue.cc:1035] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:35.292461 26225 mvcc.cc:205] Tried to move back new op lower bound from 7317642753132109824 to 7317642752980533248. Current Snapshot: MvccSnapshot[applied={T|T < 7317642753132109824}]
I20260812 11:24:35.316865 25358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:35.317191 25358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:35.321487 25358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Sending DeleteTablet for 3 replicas of tablet 6ae477a106d9488484b444b9c62a4dbd
I20260812 11:24:35.322747 25591 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae477a106d9488484b444b9c62a4dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:35 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:43696
I20260812 11:24:35.323001 26038 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae477a106d9488484b444b9c62a4dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:35 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:55490
I20260812 11:24:35.323557 25800 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ae477a106d9488484b444b9c62a4dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:35 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:35668
I20260812 11:24:35.324507 25358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:35.325646 25358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:35.327575 26250 tablet_replica.cc:330] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: stopping tablet replica
I20260812 11:24:35.328097 26250 raft_consensus.cc:2229] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:35.328805 26250 raft_consensus.cc:2258] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:35.331132 26251 tablet_replica.cc:330] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: stopping tablet replica
I20260812 11:24:35.331642 26251 raft_consensus.cc:2229] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:35.332301 26251 raft_consensus.cc:2258] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:35.332949 26249 tablet_replica.cc:330] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: stopping tablet replica
I20260812 11:24:35.333758 26249 raft_consensus.cc:2229] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:35.334743 26249 raft_consensus.cc:2258] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:35.335558 25591 tablet_service.cc:1467] Processing CreateTablet for tablet a8a38fe2288340b3b8e01a4122246e56 (DEFAULT_TABLE table=test-table [id=e221f2fe915a40e3af1c4c288e2b6a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:35.335820 26038 tablet_service.cc:1467] Processing CreateTablet for tablet a8a38fe2288340b3b8e01a4122246e56 (DEFAULT_TABLE table=test-table [id=e221f2fe915a40e3af1c4c288e2b6a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:35.336020 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a38fe2288340b3b8e01a4122246e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:35.336251 26038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a38fe2288340b3b8e01a4122246e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:35.337338 26249 ts_tablet_manager.cc:1905] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:35.339303 25800 tablet_service.cc:1467] Processing CreateTablet for tablet a8a38fe2288340b3b8e01a4122246e56 (DEFAULT_TABLE table=test-table [id=e221f2fe915a40e3af1c4c288e2b6a96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:35.339730 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a38fe2288340b3b8e01a4122246e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:35.345436 26249 ts_tablet_manager.cc:1918] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:35.345504 26250 ts_tablet_manager.cc:1905] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:35.345716 26249 log.cc:1196] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/6ae477a106d9488484b444b9c62a4dbd
I20260812 11:24:35.346405 26249 ts_tablet_manager.cc:1939] T 6ae477a106d9488484b444b9c62a4dbd P 94f63c7bf490499c8537184bf309762f: Deleting consensus metadata
I20260812 11:24:35.348824 25342 catalog_manager.cc:4841] TS 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663): tablet 6ae477a106d9488484b444b9c62a4dbd (table test-table [id=d1ff65e86e4c439bb2d5e07b562a8c76]) successfully deleted
I20260812 11:24:35.349375 26251 ts_tablet_manager.cc:1905] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:35.349412 26208 tablet_bootstrap.cc:492] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: Bootstrap starting.
I20260812 11:24:35.351377 26208 tablet_bootstrap.cc:654] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.351521 26206 tablet_bootstrap.cc:492] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: Bootstrap starting.
I20260812 11:24:35.353114 26206 tablet_bootstrap.cc:654] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.358610 26205 tablet_bootstrap.cc:492] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: Bootstrap starting.
I20260812 11:24:35.360980 26205 tablet_bootstrap.cc:654] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.363726 26206 tablet_bootstrap.cc:492] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: No bootstrap required, opened a new log
I20260812 11:24:35.363898 26206 ts_tablet_manager.cc:1397] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20260812 11:24:35.364702 26206 raft_consensus.cc:348] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.364952 26206 raft_consensus.cc:374] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.365326 26206 raft_consensus.cc:729] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f63c7bf490499c8537184bf309762f, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.365705 26206 consensus_queue.cc:260] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.368345 26251 ts_tablet_manager.cc:1918] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:35.368533 26251 log.cc:1196] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/6ae477a106d9488484b444b9c62a4dbd
I20260812 11:24:35.368719 26206 ts_tablet_manager.cc:1428] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260812 11:24:35.369145 26251 ts_tablet_manager.cc:1939] T 6ae477a106d9488484b444b9c62a4dbd P 538486cabc2d4e67a4285beb67ff43f9: Deleting consensus metadata
I20260812 11:24:35.369709 26205 tablet_bootstrap.cc:492] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: No bootstrap required, opened a new log
I20260812 11:24:35.369863 26205 ts_tablet_manager.cc:1397] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260812 11:24:35.370822 26205 raft_consensus.cc:348] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.371058 26205 raft_consensus.cc:374] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.371157 26205 raft_consensus.cc:729] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538486cabc2d4e67a4285beb67ff43f9, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.371130 25344 catalog_manager.cc:4841] TS 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249): tablet 6ae477a106d9488484b444b9c62a4dbd (table test-table [id=d1ff65e86e4c439bb2d5e07b562a8c76]) successfully deleted
I20260812 11:24:35.371711 26250 ts_tablet_manager.cc:1918] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:35.371871 26250 log.cc:1196] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/6ae477a106d9488484b444b9c62a4dbd
I20260812 11:24:35.371623 26205 consensus_queue.cc:260] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.372375 26250 ts_tablet_manager.cc:1939] T 6ae477a106d9488484b444b9c62a4dbd P 69ca1916207d48e3b21045e133391a3f: Deleting consensus metadata
I20260812 11:24:35.372474 26205 ts_tablet_manager.cc:1428] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:24:35.374033 26208 tablet_bootstrap.cc:492] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: No bootstrap required, opened a new log
I20260812 11:24:35.373996 25341 catalog_manager.cc:4841] TS 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305): tablet 6ae477a106d9488484b444b9c62a4dbd (table test-table [id=d1ff65e86e4c439bb2d5e07b562a8c76]) successfully deleted
I20260812 11:24:35.374182 26208 ts_tablet_manager.cc:1397] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.007s
I20260812 11:24:35.375036 26208 raft_consensus.cc:348] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.375300 26208 raft_consensus.cc:374] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.375419 26208 raft_consensus.cc:729] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca1916207d48e3b21045e133391a3f, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.375694 26208 consensus_queue.cc:260] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.376209 26208 ts_tablet_manager.cc:1428] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:24:35.489270 26214 raft_consensus.cc:482] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:35.489557 26214 raft_consensus.cc:504] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.491289 26214 leader_election.cc:288] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249), 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305)
I20260812 11:24:35.499607 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.491837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:33635 (local address 127.12.110.195:41305)
0812 11:24:35.492082 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.492092 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.493017 (+ 925us) server_negotiation.cc:407] Connection header received
0812 11:24:35.493269 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.493275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.493372 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.493436 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:35.494202 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.495169 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.497181 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.498476 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.498675 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.498761 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.499140 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.499295 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:24:35.499639 26259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.491679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41305 (local address 127.12.110.193:33635)
0812 11:24:35.492916 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.492933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.493121 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.493637 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.493641 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.493668 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:35.494030 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.494044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.495303 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.495309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.497016 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.497029 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.498723 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.498730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.498950 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.498989 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.499096 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.499231 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":42,"thread_start_us":880,"threads_started":1}
I20260812 11:24:35.499707 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.492022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:49181 (local address 127.12.110.194:42249)
0812 11:24:35.492204 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.492210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.492741 (+ 531us) server_negotiation.cc:407] Connection header received
0812 11:24:35.493024 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.493035 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.493096 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.493202 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:35.493718 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.494748 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.497362 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.498638 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.498777 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.498868 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.499292 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.499432 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:24:35.500552 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.491560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:42249 (local address 127.12.110.193:49181)
0812 11:24:35.492618 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.492643 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.492869 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.493229 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.493234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.493260 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:35.493549 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.493558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.494899 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.494907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.497203 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.497219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.498951 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.498956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.499111 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.499142 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.500079 (+ 937us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.500220 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":52,"thread_start_us":688,"threads_started":1}
I20260812 11:24:35.500702 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a38fe2288340b3b8e01a4122246e56" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca1916207d48e3b21045e133391a3f" is_pre_election: true
I20260812 11:24:35.501153 26073 raft_consensus.cc:2454] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 0.
I20260812 11:24:35.501216 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a38fe2288340b3b8e01a4122246e56" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9" is_pre_election: true
I20260812 11:24:35.501477 25830 raft_consensus.cc:2454] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 0.
I20260812 11:24:35.501788 25510 leader_election.cc:302] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca1916207d48e3b21045e133391a3f, 94f63c7bf490499c8537184bf309762f; no voters:
I20260812 11:24:35.502266 26214 raft_consensus.cc:2781] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:35.502390 26214 raft_consensus.cc:482] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:35.502481 26214 raft_consensus.cc:3037] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.505707 26214 raft_consensus.cc:504] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.506402 26214 leader_election.cc:288] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 election: Requested vote from peers 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249), 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305)
I20260812 11:24:35.506865 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a38fe2288340b3b8e01a4122246e56" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9"
I20260812 11:24:35.506875 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a38fe2288340b3b8e01a4122246e56" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca1916207d48e3b21045e133391a3f"
I20260812 11:24:35.507076 26073 raft_consensus.cc:3037] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.507071 25830 raft_consensus.cc:3037] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.509055 25830 raft_consensus.cc:2454] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 1.
I20260812 11:24:35.509505 25513 leader_election.cc:302] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538486cabc2d4e67a4285beb67ff43f9, 94f63c7bf490499c8537184bf309762f; no voters:
I20260812 11:24:35.509781 26214 raft_consensus.cc:2781] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:35.510311 26214 raft_consensus.cc:686] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 1 LEADER]: Becoming Leader. State: Replica: 94f63c7bf490499c8537184bf309762f, State: Running, Role: LEADER
I20260812 11:24:35.510352 26073 raft_consensus.cc:2454] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 1.
I20260812 11:24:35.510766 26214 consensus_queue.cc:237] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.514354 25358 catalog_manager.cc:5495] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f63c7bf490499c8537184bf309762f (127.12.110.193). New cstate: current_term: 1 leader_uuid: "94f63c7bf490499c8537184bf309762f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } health_report { overall_health: UNKNOWN } } }
Aug 12 11:24:35 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533872, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.254@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:35.571861 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.553478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37329 (local address 127.0.0.1:57814)
0812 11:24:35.554470 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.554502 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.554773 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.555303 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.555312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.556476 (+ 1164us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:35.556875 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.556890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.558145 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.558151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.559587 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.559605 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.560373 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.560380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.560557 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.561353 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:35.561378 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:35.564558 (+ 3180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:35.568872 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:35.568891 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:35.568899 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:35.569551 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:35.570096 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:35.570100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:35.570103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:35.570270 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:35.570982 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:35.570988 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:35.571134 (+ 146us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.571271 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.571506 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"thread_start_us":457,"threads_started":1}
I20260812 11:24:35.572122 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.553675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.12.110.254:37329)
0812 11:24:35.554773 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.554778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.554806 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:24:35.555000 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.555005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.555079 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.555194 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:35.557110 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.557939 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.559784 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.560191 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.560294 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.564792 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:35.564799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:35.564803 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:35.564840 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:35.568521 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:35.569767 (+ 1246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:35.569773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:35.569777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:35.569940 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:35.570413 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:35.570418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:35.570421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:35.570827 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:35.570998 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.571433 (+ 435us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.571666 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":969,"thread_start_us":704,"threads_started":1}
I20260812 11:24:35.577505 25358 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:57814:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260812 11:24:35.578787 25358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:35.586458 25800 tablet_service.cc:1467] Processing CreateTablet for tablet d0fa7b033d16464faafdfeca4cac3da2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5861a1b294b549e9b9ff5585719994ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:24:35.586719 25591 tablet_service.cc:1467] Processing CreateTablet for tablet d0fa7b033d16464faafdfeca4cac3da2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5861a1b294b549e9b9ff5585719994ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:24:35.587137 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fa7b033d16464faafdfeca4cac3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:35.587277 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fa7b033d16464faafdfeca4cac3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:35.591398 26038 tablet_service.cc:1467] Processing CreateTablet for tablet d0fa7b033d16464faafdfeca4cac3da2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5861a1b294b549e9b9ff5585719994ea]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260812 11:24:35.591943 26038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0fa7b033d16464faafdfeca4cac3da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:35.593048 26206 tablet_bootstrap.cc:492] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: Bootstrap starting.
I20260812 11:24:35.594810 26206 tablet_bootstrap.cc:654] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.597137 26206 tablet_bootstrap.cc:492] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: No bootstrap required, opened a new log
I20260812 11:24:35.597298 26206 ts_tablet_manager.cc:1397] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260812 11:24:35.597869 26208 tablet_bootstrap.cc:492] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: Bootstrap starting.
I20260812 11:24:35.598047 26206 raft_consensus.cc:348] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.598304 26206 raft_consensus.cc:374] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.598399 26206 raft_consensus.cc:729] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f63c7bf490499c8537184bf309762f, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.598716 26206 consensus_queue.cc:260] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.599153 26214 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.599428 26214 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:24:35.600260 26208 tablet_bootstrap.cc:654] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.600907 26206 ts_tablet_manager.cc:1428] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260812 11:24:35.602459 26205 tablet_bootstrap.cc:492] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: Bootstrap starting.
I20260812 11:24:35.604271 26205 tablet_bootstrap.cc:654] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:35.605187 26208 tablet_bootstrap.cc:492] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: No bootstrap required, opened a new log
I20260812 11:24:35.605389 26208 ts_tablet_manager.cc:1397] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 11:24:35.606143 26208 raft_consensus.cc:348] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.606401 26208 raft_consensus.cc:374] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.606509 26208 raft_consensus.cc:729] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca1916207d48e3b21045e133391a3f, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.606601 26205 tablet_bootstrap.cc:492] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: No bootstrap required, opened a new log
I20260812 11:24:35.606756 26205 ts_tablet_manager.cc:1397] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260812 11:24:35.606853 26208 consensus_queue.cc:260] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.607453 26208 ts_tablet_manager.cc:1428] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260812 11:24:35.607301 26220 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.607681 26220 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:24:35.608014 26205 raft_consensus.cc:348] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.608268 26205 raft_consensus.cc:374] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:35.608346 26205 raft_consensus.cc:729] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538486cabc2d4e67a4285beb67ff43f9, State: Initialized, Role: FOLLOWER
I20260812 11:24:35.608589 26205 consensus_queue.cc:260] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.609051 26213 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.609418 26213 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:24:35.609869 26205 ts_tablet_manager.cc:1428] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260812 11:24:35.622310 26220 raft_consensus.cc:482] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:35.622558 26220 raft_consensus.cc:504] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.623286 26220 leader_election.cc:288] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663), 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249)
I20260812 11:24:35.623769 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa7b033d16464faafdfeca4cac3da2" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9" is_pre_election: true
I20260812 11:24:35.623818 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa7b033d16464faafdfeca4cac3da2" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f63c7bf490499c8537184bf309762f" is_pre_election: true
I20260812 11:24:35.624044 25830 raft_consensus.cc:2454] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 0.
I20260812 11:24:35.624089 25620 raft_consensus.cc:2454] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 0.
I20260812 11:24:35.624450 25946 leader_election.cc:302] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538486cabc2d4e67a4285beb67ff43f9, 69ca1916207d48e3b21045e133391a3f; no voters:
I20260812 11:24:35.624783 26220 raft_consensus.cc:2781] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:35.624904 26220 raft_consensus.cc:482] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:35.625032 26220 raft_consensus.cc:3037] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.627446 26220 raft_consensus.cc:504] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.628093 26220 leader_election.cc:288] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 election: Requested vote from peers 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663), 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249)
I20260812 11:24:35.628511 25620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa7b033d16464faafdfeca4cac3da2" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f63c7bf490499c8537184bf309762f"
I20260812 11:24:35.628582 25830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0fa7b033d16464faafdfeca4cac3da2" candidate_uuid: "69ca1916207d48e3b21045e133391a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9"
I20260812 11:24:35.628748 25620 raft_consensus.cc:3037] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.628814 25830 raft_consensus.cc:3037] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:35.631788 25830 raft_consensus.cc:2454] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 1.
I20260812 11:24:35.631806 25620 raft_consensus.cc:2454] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ca1916207d48e3b21045e133391a3f in term 1.
I20260812 11:24:35.632226 25945 leader_election.cc:302] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 538486cabc2d4e67a4285beb67ff43f9, 69ca1916207d48e3b21045e133391a3f, 94f63c7bf490499c8537184bf309762f; no voters:
I20260812 11:24:35.632705 26220 raft_consensus.cc:2781] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:35.632952 26220 raft_consensus.cc:686] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [term 1 LEADER]: Becoming Leader. State: Replica: 69ca1916207d48e3b21045e133391a3f, State: Running, Role: LEADER
I20260812 11:24:35.633246 26220 consensus_queue.cc:237] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:35.635174 26236 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: TxnStatusTablet state changed. Reason: New leader 69ca1916207d48e3b21045e133391a3f. Latest consensus state: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.635664 26236 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:24:35.636528 25358 catalog_manager.cc:5495] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ca1916207d48e3b21045e133391a3f (127.12.110.195). New cstate: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } health_report { overall_health: HEALTHY } } }
I20260812 11:24:35.637794 26277 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260812 11:24:35.638048 26277 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260812 11:24:35.659379 26269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.652375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41305 (local address 127.0.0.1:55508)
0812 11:24:35.652802 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.652824 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.652980 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.654499 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.654506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.654552 (+ 46us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:24:35.654741 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.654750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.656247 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.656254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.657274 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.657286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.658099 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.658104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.658275 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.658331 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.658776 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.658802 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.658912 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.659071 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260812 11:24:35.659356 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.653677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.12.110.195:41305)
0812 11:24:35.653927 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.653936 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.653960 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:24:35.654039 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.654047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.654161 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.654265 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:35.654893 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.656104 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.657431 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.657920 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.658087 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.658523 (+ 436us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.658663 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.658778 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.658940 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.659030 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:24:35.663547 25620 raft_consensus.cc:1261] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Refusing update from remote peer 69ca1916207d48e3b21045e133391a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:35.663694 25830 raft_consensus.cc:1261] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Refusing update from remote peer 69ca1916207d48e3b21045e133391a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:35.664244 26236 consensus_queue.cc:1035] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:35.664565 26220 consensus_queue.cc:1035] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:35.671464 26213 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: TxnStatusTablet state changed. Reason: New leader 69ca1916207d48e3b21045e133391a3f. Latest consensus state: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.671799 26213 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:24:35.673178 26214 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: TxnStatusTablet state changed. Reason: New leader 69ca1916207d48e3b21045e133391a3f. Latest consensus state: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.673502 26214 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:24:35.676080 26220 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.676407 26220 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:24:35.676676 26220 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.676934 26220 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 69ca1916207d48e3b21045e133391a3f: This TxnStatusTablet replica's current role is: LEADER
I20260812 11:24:35.678313 26213 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.678572 26213 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 538486cabc2d4e67a4285beb67ff43f9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:24:35.681907 26214 tablet_replica.cc:439] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69ca1916207d48e3b21045e133391a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } } }
I20260812 11:24:35.682193 26214 tablet_replica.cc:442] T d0fa7b033d16464faafdfeca4cac3da2 P 94f63c7bf490499c8537184bf309762f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260812 11:24:35.690165 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.683300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43706 (local address 127.12.110.193:40663)
0812 11:24:35.683797 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.683801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.683816 (+ 15us) server_negotiation.cc:407] Connection header received
0812 11:24:35.683906 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.683909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.684019 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.684136 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:35.684599 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.685773 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.687820 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.688224 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.688328 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.688939 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.689476 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.689582 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.689762 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.689828 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406}
I20260812 11:24:35.708741 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.702300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:60577 (local address 127.12.110.195:41305)
0812 11:24:35.702624 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.702634 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.703292 (+ 658us) server_negotiation.cc:407] Connection header received
0812 11:24:35.703535 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.703539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.703605 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.703718 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:35.704388 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.705190 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.706655 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.707121 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.707266 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.707854 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.708011 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.708101 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.708320 (+ 219us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.708406 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 11:24:35.708853 26290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.702347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41305 (local address 127.12.110.193:60577)
0812 11:24:35.703186 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.703202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.703385 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.703757 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.703761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.703775 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:24:35.704201 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.704215 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.705338 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.705344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.706486 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.706498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.707311 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.707320 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.707549 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.707619 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.708186 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.708196 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.708293 (+ 97us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.708430 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":481,"threads_started":1}
I20260812 11:24:35.720551 26073 raft_consensus.cc:1261] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Refusing update from remote peer 94f63c7bf490499c8537184bf309762f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:35.721303 25830 raft_consensus.cc:1261] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Refusing update from remote peer 94f63c7bf490499c8537184bf309762f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:35.721407 26260 consensus_queue.cc:1035] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:35.721870 26214 consensus_queue.cc:1035] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [LEADER]: Connected to new peer: Peer: permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:35.734313 26227 mvcc.cc:205] Tried to move back new op lower bound from 7317642754943664128 to 7317642754096226304. Current Snapshot: MvccSnapshot[applied={T|T < 7317642754943664128}]
I20260812 11:24:35.736136 26224 mvcc.cc:205] Tried to move back new op lower bound from 7317642754943664128 to 7317642754096226304. Current Snapshot: MvccSnapshot[applied={T|T < 7317642754943664128}]
I20260812 11:24:35.789077 26297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.780436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:40663 (local address 127.12.110.195:59701)
0812 11:24:35.781170 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:35.781192 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:35.781353 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:35.783105 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:35.783110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:35.783124 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0812 11:24:35.783496 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.783507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.785011 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.785017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.786867 (+ 1850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:35.786880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.787917 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.787922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:35.788047 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.788135 (+ 88us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.788595 (+ 460us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.788604 (+ 9us) client_negotiation.cc:769] Sending connection context
0812 11:24:35.788665 (+ 61us) client_negotiation.cc:240] Negotiation successful
0812 11:24:35.788768 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":1140,"threads_started":1}
I20260812 11:24:35.789101 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.780425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:59701 (local address 127.12.110.193:40663)
0812 11:24:35.780661 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.780668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.781300 (+ 632us) server_negotiation.cc:407] Connection header received
0812 11:24:35.782779 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.782785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.782881 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.782998 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:35.783706 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.784843 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.787012 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.787723 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.787868 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.788335 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:35.788485 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:35.788589 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.788725 (+ 136us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.788807 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:24:35.843089 25358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:35.843379 25358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:35.847265 25358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Sending DeleteTablet for 3 replicas of tablet a8a38fe2288340b3b8e01a4122246e56
I20260812 11:24:35.848729 25800 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a38fe2288340b3b8e01a4122246e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:35 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:35668
I20260812 11:24:35.849099 26251 tablet_replica.cc:330] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: stopping tablet replica
I20260812 11:24:35.849223 26038 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a38fe2288340b3b8e01a4122246e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:35 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:55490
I20260812 11:24:35.849306 26251 raft_consensus.cc:2229] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:35.849496 26251 raft_consensus.cc:2258] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:35.849555 25591 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a38fe2288340b3b8e01a4122246e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:35 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:43696
I20260812 11:24:35.849838 26250 tablet_replica.cc:330] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: stopping tablet replica
I20260812 11:24:35.850030 26250 raft_consensus.cc:2229] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:35.850251 26250 raft_consensus.cc:2258] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:35.852000 26250 ts_tablet_manager.cc:1905] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:35.853107 26251 ts_tablet_manager.cc:1905] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:35.862282 26250 ts_tablet_manager.cc:1918] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:24:35.867278 26250 log.cc:1196] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/a8a38fe2288340b3b8e01a4122246e56
I20260812 11:24:35.867970 26250 ts_tablet_manager.cc:1939] T a8a38fe2288340b3b8e01a4122246e56 P 69ca1916207d48e3b21045e133391a3f: Deleting consensus metadata
I20260812 11:24:35.868649 26251 ts_tablet_manager.cc:1918] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:24:35.868885 26251 log.cc:1196] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/a8a38fe2288340b3b8e01a4122246e56
I20260812 11:24:35.869414 26251 ts_tablet_manager.cc:1939] T a8a38fe2288340b3b8e01a4122246e56 P 538486cabc2d4e67a4285beb67ff43f9: Deleting consensus metadata
I20260812 11:24:35.872259 25341 catalog_manager.cc:4841] TS 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305): tablet a8a38fe2288340b3b8e01a4122246e56 (table test-table [id=e221f2fe915a40e3af1c4c288e2b6a96]) successfully deleted
I20260812 11:24:35.873548 25344 catalog_manager.cc:4841] TS 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249): tablet a8a38fe2288340b3b8e01a4122246e56 (table test-table [id=e221f2fe915a40e3af1c4c288e2b6a96]) successfully deleted
I20260812 11:24:35.887104 26300 tablet_replica.cc:330] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: stopping tablet replica
I20260812 11:24:35.898963 26300 raft_consensus.cc:2229] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:35.899639 26300 raft_consensus.cc:2258] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:35.901292 26300 ts_tablet_manager.cc:1905] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:35.910003 26300 ts_tablet_manager.cc:1918] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:24:35.910173 26300 log.cc:1196] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/a8a38fe2288340b3b8e01a4122246e56
I20260812 11:24:35.910665 26300 ts_tablet_manager.cc:1939] T a8a38fe2288340b3b8e01a4122246e56 P 94f63c7bf490499c8537184bf309762f: Deleting consensus metadata
I20260812 11:24:35.911968 25342 catalog_manager.cc:4841] TS 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663): tablet a8a38fe2288340b3b8e01a4122246e56 (table test-table [id=e221f2fe915a40e3af1c4c288e2b6a96]) successfully deleted
Aug 12 11:24:35 dist-test-slave-h6n0 krb5kdc[25286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 12 11:24:35 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:35 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:24:35.966336 26271 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:35.953257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.12.110.254:37329)
0812 11:24:35.953471 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.953479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.953927 (+ 448us) server_negotiation.cc:407] Connection header received
0812 11:24:35.954137 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.954143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.954245 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.954363 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:35.956125 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.957287 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.959242 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.959907 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.960044 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.966058 (+ 6014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57820: BlockingRecv error: recv got EOF from 127.0.0.1:57820 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 12 11:24:35 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM
I20260812 11:24:35.990082 26271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:35.971852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57824 (local address 127.12.110.254:37329)
0812 11:24:35.972040 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:35.972046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:35.972633 (+ 587us) server_negotiation.cc:407] Connection header received
0812 11:24:35.972786 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:35.972792 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:35.972868 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:35.972979 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:35.974572 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.975715 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.977716 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:35.978428 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:35.978580 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:35.983439 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:35.983446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:35.983472 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:35.983523 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:35.987044 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:35.987849 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:35.987855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:35.987859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:35.988027 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:35.988493 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:35.988499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:35.988510 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:35.988873 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:35.989142 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:35.989456 (+ 314us) server_negotiation.cc:299] Negotiation successful
0812 11:24:35.989708 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:24:35.994340 25358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:35.995124 25358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:36.003883 25800 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4fe64e10b245a3a839ce2bc84d8006 (DEFAULT_TABLE table=test-table [id=83ea37f694d3422ba2803bafdd2e2c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:36.004133 25591 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4fe64e10b245a3a839ce2bc84d8006 (DEFAULT_TABLE table=test-table [id=83ea37f694d3422ba2803bafdd2e2c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:36.004206 26038 tablet_service.cc:1467] Processing CreateTablet for tablet 7c4fe64e10b245a3a839ce2bc84d8006 (DEFAULT_TABLE table=test-table [id=83ea37f694d3422ba2803bafdd2e2c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:36.004384 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4fe64e10b245a3a839ce2bc84d8006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:36.004567 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4fe64e10b245a3a839ce2bc84d8006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:36.004648 26038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c4fe64e10b245a3a839ce2bc84d8006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:36.010836 26205 tablet_bootstrap.cc:492] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: Bootstrap starting.
I20260812 11:24:36.011850 26206 tablet_bootstrap.cc:492] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: Bootstrap starting.
I20260812 11:24:36.012517 26205 tablet_bootstrap.cc:654] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:36.012791 26208 tablet_bootstrap.cc:492] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: Bootstrap starting.
I20260812 11:24:36.013495 26206 tablet_bootstrap.cc:654] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:36.015336 26208 tablet_bootstrap.cc:654] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:36.020488 26206 tablet_bootstrap.cc:492] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: No bootstrap required, opened a new log
I20260812 11:24:36.020665 26206 ts_tablet_manager.cc:1397] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260812 11:24:36.020695 26208 tablet_bootstrap.cc:492] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: No bootstrap required, opened a new log
I20260812 11:24:36.020828 26208 ts_tablet_manager.cc:1397] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20260812 11:24:36.021027 26205 tablet_bootstrap.cc:492] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: No bootstrap required, opened a new log
I20260812 11:24:36.021230 26205 ts_tablet_manager.cc:1397] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260812 11:24:36.021622 26206 raft_consensus.cc:348] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.021924 26206 raft_consensus.cc:374] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:36.021627 26208 raft_consensus.cc:348] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.022042 26206 raft_consensus.cc:729] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f63c7bf490499c8537184bf309762f, State: Initialized, Role: FOLLOWER
I20260812 11:24:36.022065 26208 raft_consensus.cc:374] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:36.022176 26208 raft_consensus.cc:729] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca1916207d48e3b21045e133391a3f, State: Initialized, Role: FOLLOWER
I20260812 11:24:36.022292 26206 consensus_queue.cc:260] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.022459 26208 consensus_queue.cc:260] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.022964 26206 ts_tablet_manager.cc:1428] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260812 11:24:36.023075 26208 ts_tablet_manager.cc:1428] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260812 11:24:36.022451 26205 raft_consensus.cc:348] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.023475 26205 raft_consensus.cc:374] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:36.023634 26205 raft_consensus.cc:729] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538486cabc2d4e67a4285beb67ff43f9, State: Initialized, Role: FOLLOWER
I20260812 11:24:36.023988 26205 consensus_queue.cc:260] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.025061 26205 ts_tablet_manager.cc:1428] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260812 11:24:36.086772 26213 raft_consensus.cc:482] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:36.087028 26213 raft_consensus.cc:504] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.088915 26213 leader_election.cc:288] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663), 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305)
I20260812 11:24:36.109572 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.089312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:37069 (local address 127.12.110.193:40663)
0812 11:24:36.089540 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:36.089548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:36.101612 (+ 12064us) server_negotiation.cc:407] Connection header received
0812 11:24:36.102102 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:36.102108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:36.102189 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:36.102255 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:36.103115 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.104334 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.106984 (+ 2650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.108294 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.108443 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.108518 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:36.109082 (+ 564us) server_negotiation.cc:299] Negotiation successful
0812 11:24:36.109229 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:24:36.110675 26330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.089097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:40663 (local address 127.12.110.194:37069)
0812 11:24:36.101479 (+ 12382us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:36.101504 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:36.101898 (+ 394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:36.102457 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:36.102462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:36.102500 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:36.102887 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:36.102900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.104582 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.104586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:36.106775 (+ 2189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:36.106791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.108666 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.108672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:36.108855 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.108893 (+ 38us) client_negotiation.cc:769] Sending connection context
0812 11:24:36.110079 (+ 1186us) client_negotiation.cc:240] Negotiation successful
0812 11:24:36.110233 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12230,"thread_start_us":16614,"threads_started":1}
I20260812 11:24:36.111490 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4fe64e10b245a3a839ce2bc84d8006" candidate_uuid: "538486cabc2d4e67a4285beb67ff43f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f63c7bf490499c8537184bf309762f" is_pre_election: true
I20260812 11:24:36.111766 25619 raft_consensus.cc:2454] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 538486cabc2d4e67a4285beb67ff43f9 in term 0.
I20260812 11:24:36.112458 25738 leader_election.cc:302] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538486cabc2d4e67a4285beb67ff43f9, 94f63c7bf490499c8537184bf309762f; no voters:
I20260812 11:24:36.112989 26213 raft_consensus.cc:2781] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:36.113111 26213 raft_consensus.cc:482] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:36.113204 26213 raft_consensus.cc:3037] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:36.113852 26204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.090337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:55279 (local address 127.12.110.195:41305)
0812 11:24:36.090511 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:36.090516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:36.105697 (+ 15181us) server_negotiation.cc:407] Connection header received
0812 11:24:36.106306 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:36.106313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:36.106432 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:36.106503 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:24:36.107547 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.108679 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.111098 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.112479 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.112606 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.112683 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:36.113449 (+ 766us) server_negotiation.cc:299] Negotiation successful
0812 11:24:36.113600 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260812 11:24:36.114679 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.089904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:41305 (local address 127.12.110.194:55279)
0812 11:24:36.105594 (+ 15690us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:36.105613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:36.105913 (+ 300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:36.106966 (+ 1053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:36.106971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:36.106999 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:24:36.107342 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:36.107354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.108861 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.108866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:36.110923 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:36.110939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.112889 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.112896 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:36.113088 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.113124 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:24:36.114093 (+ 969us) client_negotiation.cc:240] Negotiation successful
0812 11:24:36.114258 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15554,"thread_start_us":11728,"threads_started":1}
I20260812 11:24:36.115315 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4fe64e10b245a3a839ce2bc84d8006" candidate_uuid: "538486cabc2d4e67a4285beb67ff43f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca1916207d48e3b21045e133391a3f" is_pre_election: true
I20260812 11:24:36.115567 26073 raft_consensus.cc:2454] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 538486cabc2d4e67a4285beb67ff43f9 in term 0.
I20260812 11:24:36.116997 26213 raft_consensus.cc:504] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.117723 26213 leader_election.cc:288] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [CANDIDATE]: Term 1 election: Requested vote from peers 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663), 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305)
I20260812 11:24:36.118268 25619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4fe64e10b245a3a839ce2bc84d8006" candidate_uuid: "538486cabc2d4e67a4285beb67ff43f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "94f63c7bf490499c8537184bf309762f"
I20260812 11:24:36.118350 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c4fe64e10b245a3a839ce2bc84d8006" candidate_uuid: "538486cabc2d4e67a4285beb67ff43f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca1916207d48e3b21045e133391a3f"
I20260812 11:24:36.118584 26073 raft_consensus.cc:3037] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:36.118515 25619 raft_consensus.cc:3037] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:36.121757 26073 raft_consensus.cc:2454] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538486cabc2d4e67a4285beb67ff43f9 in term 1.
I20260812 11:24:36.121831 25619 raft_consensus.cc:2454] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 538486cabc2d4e67a4285beb67ff43f9 in term 1.
I20260812 11:24:36.122181 25736 leader_election.cc:302] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538486cabc2d4e67a4285beb67ff43f9, 69ca1916207d48e3b21045e133391a3f; no voters:
I20260812 11:24:36.122567 26213 raft_consensus.cc:2781] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:36.124217 26213 raft_consensus.cc:686] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 LEADER]: Becoming Leader. State: Replica: 538486cabc2d4e67a4285beb67ff43f9, State: Running, Role: LEADER
I20260812 11:24:36.124686 26213 consensus_queue.cc:237] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.128551 25358 catalog_manager.cc:5495] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194). New cstate: current_term: 1 leader_uuid: "538486cabc2d4e67a4285beb67ff43f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } health_report { overall_health: UNKNOWN } } }
I20260812 11:24:36.162025 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.154041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.12.110.194:42249)
0812 11:24:36.154656 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:36.154663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:36.154688 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:24:36.154800 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:36.154806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:36.155158 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:36.155491 (+ 333us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:36.155930 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.157112 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.159016 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.159682 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.160343 (+ 661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.160500 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:36.161256 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:36.161426 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:36.161550 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 11:24:36.161639 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":291,"threads_started":1}
I20260812 11:24:36.165887 26073 raft_consensus.cc:1261] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Refusing update from remote peer 538486cabc2d4e67a4285beb67ff43f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:36.165975 25619 raft_consensus.cc:1261] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Refusing update from remote peer 538486cabc2d4e67a4285beb67ff43f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:36.166778 26213 consensus_queue.cc:1035] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:36.167382 26213 consensus_queue.cc:1035] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:36.207440 25358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:36.207706 25358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:36.211992 25358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Sending DeleteTablet for 3 replicas of tablet 7c4fe64e10b245a3a839ce2bc84d8006
I20260812 11:24:36.213056 25591 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4fe64e10b245a3a839ce2bc84d8006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:36 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:43696
I20260812 11:24:36.213032 26038 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4fe64e10b245a3a839ce2bc84d8006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:36 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:55490
I20260812 11:24:36.213335 26300 tablet_replica.cc:330] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: stopping tablet replica
I20260812 11:24:36.213363 26250 tablet_replica.cc:330] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: stopping tablet replica
I20260812 11:24:36.213533 26300 raft_consensus.cc:2229] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:36.213649 25800 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c4fe64e10b245a3a839ce2bc84d8006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:36 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:35668
I20260812 11:24:36.213747 26300 raft_consensus.cc:2258] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:36.213728 26250 raft_consensus.cc:2229] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:36.213928 26251 tablet_replica.cc:330] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: stopping tablet replica
I20260812 11:24:36.213999 26250 raft_consensus.cc:2258] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:36.214103 26251 raft_consensus.cc:2229] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:36.215037 26250 ts_tablet_manager.cc:1905] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:36.216279 26251 raft_consensus.cc:2258] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:36.216774 25358 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:24:36.217454 25358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:24:36.217737 26251 ts_tablet_manager.cc:1905] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:36.219216 26300 ts_tablet_manager.cc:1905] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:36.229041 26251 ts_tablet_manager.cc:1918] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:36.229202 26251 log.cc:1196] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/7c4fe64e10b245a3a839ce2bc84d8006
I20260812 11:24:36.229679 26251 ts_tablet_manager.cc:1939] T 7c4fe64e10b245a3a839ce2bc84d8006 P 538486cabc2d4e67a4285beb67ff43f9: Deleting consensus metadata
I20260812 11:24:36.230974 25344 catalog_manager.cc:4841] TS 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249): tablet 7c4fe64e10b245a3a839ce2bc84d8006 (table test-table [id=83ea37f694d3422ba2803bafdd2e2c7b]) successfully deleted
I20260812 11:24:36.232796 26250 ts_tablet_manager.cc:1918] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:36.233054 26250 log.cc:1196] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/7c4fe64e10b245a3a839ce2bc84d8006
I20260812 11:24:36.234189 26038 tablet_service.cc:1467] Processing CreateTablet for tablet 8b07f98a52894b49b486b9b220b9015e (DEFAULT_TABLE table=test-table [id=16930c257f454c63ba1e5882202a1fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:36.234373 25591 tablet_service.cc:1467] Processing CreateTablet for tablet 8b07f98a52894b49b486b9b220b9015e (DEFAULT_TABLE table=test-table [id=16930c257f454c63ba1e5882202a1fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:36.235060 25591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b07f98a52894b49b486b9b220b9015e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:36.235028 26038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b07f98a52894b49b486b9b220b9015e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:36.239718 25800 tablet_service.cc:1467] Processing CreateTablet for tablet 8b07f98a52894b49b486b9b220b9015e (DEFAULT_TABLE table=test-table [id=16930c257f454c63ba1e5882202a1fbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:24:36.240226 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b07f98a52894b49b486b9b220b9015e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:24:36.246919 26300 ts_tablet_manager.cc:1918] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:24:36.247239 26206 tablet_bootstrap.cc:492] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: Bootstrap starting.
I20260812 11:24:36.249289 26206 tablet_bootstrap.cc:654] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:36.252086 26205 tablet_bootstrap.cc:492] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: Bootstrap starting.
I20260812 11:24:36.252466 26300 log.cc:1196] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/7c4fe64e10b245a3a839ce2bc84d8006
I20260812 11:24:36.253248 26300 ts_tablet_manager.cc:1939] T 7c4fe64e10b245a3a839ce2bc84d8006 P 94f63c7bf490499c8537184bf309762f: Deleting consensus metadata
I20260812 11:24:36.253777 26205 tablet_bootstrap.cc:654] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:36.255016 25342 catalog_manager.cc:4841] TS 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663): tablet 7c4fe64e10b245a3a839ce2bc84d8006 (table test-table [id=83ea37f694d3422ba2803bafdd2e2c7b]) successfully deleted
I20260812 11:24:36.258437 26208 tablet_bootstrap.cc:492] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: Bootstrap starting.
I20260812 11:24:36.260308 26206 tablet_bootstrap.cc:492] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: No bootstrap required, opened a new log
I20260812 11:24:36.260385 26208 tablet_bootstrap.cc:654] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:36.260499 26206 ts_tablet_manager.cc:1397] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20260812 11:24:36.261390 26206 raft_consensus.cc:348] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.261740 26206 raft_consensus.cc:374] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:36.261838 26206 raft_consensus.cc:729] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f63c7bf490499c8537184bf309762f, State: Initialized, Role: FOLLOWER
I20260812 11:24:36.262172 26206 consensus_queue.cc:260] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.264724 26206 ts_tablet_manager.cc:1428] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260812 11:24:36.289050 26250 ts_tablet_manager.cc:1939] T 7c4fe64e10b245a3a839ce2bc84d8006 P 69ca1916207d48e3b21045e133391a3f: Deleting consensus metadata
I20260812 11:24:36.296021 25341 catalog_manager.cc:4841] TS 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305): tablet 7c4fe64e10b245a3a839ce2bc84d8006 (table test-table [id=83ea37f694d3422ba2803bafdd2e2c7b]) successfully deleted
I20260812 11:24:36.311893 26208 tablet_bootstrap.cc:492] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: No bootstrap required, opened a new log
I20260812 11:24:36.312104 26208 ts_tablet_manager.cc:1397] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.004s
I20260812 11:24:36.313032 26208 raft_consensus.cc:348] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.313329 26208 raft_consensus.cc:374] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:36.313422 26208 raft_consensus.cc:729] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ca1916207d48e3b21045e133391a3f, State: Initialized, Role: FOLLOWER
I20260812 11:24:36.313687 26208 consensus_queue.cc:260] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.314247 26208 ts_tablet_manager.cc:1428] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260812 11:24:36.317289 26205 tablet_bootstrap.cc:492] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: No bootstrap required, opened a new log
I20260812 11:24:36.317518 26205 ts_tablet_manager.cc:1397] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: Time spent bootstrapping tablet: real 0.065s user 0.009s sys 0.003s
I20260812 11:24:36.318732 26205 raft_consensus.cc:348] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.319041 26205 raft_consensus.cc:374] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:36.319203 26205 raft_consensus.cc:729] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538486cabc2d4e67a4285beb67ff43f9, State: Initialized, Role: FOLLOWER
I20260812 11:24:36.322356 26205 consensus_queue.cc:260] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.326365 26205 ts_tablet_manager.cc:1428] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260812 11:24:36.442049 26260 raft_consensus.cc:482] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:24:36.442312 26260 raft_consensus.cc:504] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.443048 26260 leader_election.cc:288] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249), 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305)
I20260812 11:24:36.443802 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07f98a52894b49b486b9b220b9015e" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca1916207d48e3b21045e133391a3f" is_pre_election: true
I20260812 11:24:36.444075 26073 raft_consensus.cc:2454] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 0.
I20260812 11:24:36.444609 25510 leader_election.cc:302] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ca1916207d48e3b21045e133391a3f, 94f63c7bf490499c8537184bf309762f; no voters:
I20260812 11:24:36.445428 25829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07f98a52894b49b486b9b220b9015e" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9" is_pre_election: true
I20260812 11:24:36.445746 25829 raft_consensus.cc:2454] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 0.
I20260812 11:24:36.446682 26260 raft_consensus.cc:2781] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:24:36.446791 26260 raft_consensus.cc:482] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:24:36.446913 26260 raft_consensus.cc:3037] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:36.450161 26260 raft_consensus.cc:504] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.451284 25829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07f98a52894b49b486b9b220b9015e" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538486cabc2d4e67a4285beb67ff43f9"
I20260812 11:24:36.451504 25829 raft_consensus.cc:3037] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:36.455336 25829 raft_consensus.cc:2454] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 1.
I20260812 11:24:36.455817 25513 leader_election.cc:302] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538486cabc2d4e67a4285beb67ff43f9, 94f63c7bf490499c8537184bf309762f; no voters:
I20260812 11:24:36.457481 26073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b07f98a52894b49b486b9b220b9015e" candidate_uuid: "94f63c7bf490499c8537184bf309762f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ca1916207d48e3b21045e133391a3f"
I20260812 11:24:36.457723 26073 raft_consensus.cc:3037] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:36.460928 26073 raft_consensus.cc:2454] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94f63c7bf490499c8537184bf309762f in term 1.
I20260812 11:24:36.461678 26260 leader_election.cc:288] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [CANDIDATE]: Term 1 election: Requested vote from peers 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249), 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305)
I20260812 11:24:36.461884 26260 raft_consensus.cc:2781] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:36.462182 26260 raft_consensus.cc:686] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 1 LEADER]: Becoming Leader. State: Replica: 94f63c7bf490499c8537184bf309762f, State: Running, Role: LEADER
I20260812 11:24:36.462489 26260 consensus_queue.cc:237] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } }
I20260812 11:24:36.472843 25357 catalog_manager.cc:5495] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f reported cstate change: term changed from 0 to 1, leader changed from <none> to 94f63c7bf490499c8537184bf309762f (127.12.110.193). New cstate: current_term: 1 leader_uuid: "94f63c7bf490499c8537184bf309762f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94f63c7bf490499c8537184bf309762f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 40663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 } health_report { overall_health: UNKNOWN } } }
I20260812 11:24:36.513295 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.506679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43720 (local address 127.12.110.193:40663)
0812 11:24:36.506834 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:36.506839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:36.506855 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:24:36.506911 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:36.506914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:36.506975 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:36.507055 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:24:36.507566 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.508760 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.510805 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.511437 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.511562 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.512175 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:24:36.512318 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:24:36.512413 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:36.512915 (+ 502us) server_negotiation.cc:299] Negotiation successful
0812 11:24:36.512996 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260812 11:24:36.526499 26073 raft_consensus.cc:1261] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Refusing update from remote peer 94f63c7bf490499c8537184bf309762f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:36.526876 25829 raft_consensus.cc:1261] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Refusing update from remote peer 94f63c7bf490499c8537184bf309762f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:24:36.527159 26345 consensus_queue.cc:1035] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ca1916207d48e3b21045e133391a3f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:36.527738 26260 consensus_queue.cc:1035] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [LEADER]: Connected to new peer: Peer: permanent_uuid: "538486cabc2d4e67a4285beb67ff43f9" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:24:36.602119 25357 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:36.602356 25357 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:24:36.606029 25357 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b25772c68b764fe4babb66b80064dd67: Sending DeleteTablet for 3 replicas of tablet 8b07f98a52894b49b486b9b220b9015e
I20260812 11:24:36.606956 25591 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b07f98a52894b49b486b9b220b9015e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:36 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:43696
I20260812 11:24:36.606927 25800 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b07f98a52894b49b486b9b220b9015e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:36 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:35668
I20260812 11:24:36.607019 26038 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b07f98a52894b49b486b9b220b9015e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:24:36 UTC) from {username='oryx', principal='oryx/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:55490
I20260812 11:24:36.607250 26251 tablet_replica.cc:330] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: stopping tablet replica
I20260812 11:24:36.607256 26300 tablet_replica.cc:330] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: stopping tablet replica
I20260812 11:24:36.607316 26250 tablet_replica.cc:330] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: stopping tablet replica
I20260812 11:24:36.607430 26300 raft_consensus.cc:2229] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:24:36.607460 26251 raft_consensus.cc:2229] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:36.607522 26250 raft_consensus.cc:2229] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:24:36.607650 26251 raft_consensus.cc:2258] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:36.607719 26250 raft_consensus.cc:2258] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:36.607808 26300 raft_consensus.cc:2258] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:24:36.608971 26250 ts_tablet_manager.cc:1905] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:36.608965 26251 ts_tablet_manager.cc:1905] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:36.608984 26300 ts_tablet_manager.cc:1905] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:24:36.615933 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 25471
I20260812 11:24:36.615940 26300 ts_tablet_manager.cc:1918] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:24:36.616006 26251 ts_tablet_manager.cc:1918] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:24:36.616086 26300 log.cc:1196] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/8b07f98a52894b49b486b9b220b9015e
I20260812 11:24:36.616161 26251 log.cc:1196] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/8b07f98a52894b49b486b9b220b9015e
I20260812 11:24:36.616575 26300 ts_tablet_manager.cc:1939] T 8b07f98a52894b49b486b9b220b9015e P 94f63c7bf490499c8537184bf309762f: Deleting consensus metadata
I20260812 11:24:36.616613 26251 ts_tablet_manager.cc:1939] T 8b07f98a52894b49b486b9b220b9015e P 538486cabc2d4e67a4285beb67ff43f9: Deleting consensus metadata
I20260812 11:24:36.617713 25342 catalog_manager.cc:4841] TS 94f63c7bf490499c8537184bf309762f (127.12.110.193:40663): tablet 8b07f98a52894b49b486b9b220b9015e (table test-table [id=16930c257f454c63ba1e5882202a1fbb]) successfully deleted
I20260812 11:24:36.618119 25344 catalog_manager.cc:4841] TS 538486cabc2d4e67a4285beb67ff43f9 (127.12.110.194:42249): tablet 8b07f98a52894b49b486b9b220b9015e (table test-table [id=16930c257f454c63ba1e5882202a1fbb]) successfully deleted
I20260812 11:24:36.620795 26250 ts_tablet_manager.cc:1918] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260812 11:24:36.621021 26250 log.cc:1196] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/8b07f98a52894b49b486b9b220b9015e
I20260812 11:24:36.621537 26250 ts_tablet_manager.cc:1939] T 8b07f98a52894b49b486b9b220b9015e P 69ca1916207d48e3b21045e133391a3f: Deleting consensus metadata
I20260812 11:24:36.622947 25341 catalog_manager.cc:4841] TS 69ca1916207d48e3b21045e133391a3f (127.12.110.195:41305): tablet 8b07f98a52894b49b486b9b220b9015e (table test-table [id=16930c257f454c63ba1e5882202a1fbb]) successfully deleted
Aug 12 11:24:36 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.193@KRBTEST.COM
I20260812 11:24:36.640992 26202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.619362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43726 (local address 127.12.110.193:40663)
0812 11:24:36.619557 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:36.619566 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:36.623101 (+ 3535us) server_negotiation.cc:407] Connection header received
0812 11:24:36.623338 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:36.623343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:36.623408 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:36.623568 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:36.625454 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.626285 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.628298 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.628703 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.628805 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.634353 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:36.634370 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:36.634386 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:36.634425 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:36.637832 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:36.638652 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:36.638660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:36.638662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:36.638801 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:36.639232 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:36.639237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:36.639239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:36.639634 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:36.640020 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:36.640361 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 11:24:36.640612 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:24:36.641892 25697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:36.836472 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 25471
I20260812 11:24:36.867940 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 25710
Aug 12 11:24:36 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.194@KRBTEST.COM
I20260812 11:24:36.887042 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:36.868882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.12.110.194:42249)
0812 11:24:36.869415 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:36.869420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:36.869438 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:24:36.869496 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:36.869500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:36.869580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:36.869714 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:36.871508 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.872341 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.874344 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:36.874995 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:36.875106 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:36.880293 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:36.880311 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:36.880328 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:36.880374 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:36.883697 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:36.884625 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:36.884632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:36.884635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:36.884787 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:36.885215 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:36.885222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:36.885225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:36.885653 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:36.886012 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:36.886344 (+ 332us) server_negotiation.cc:299] Negotiation successful
0812 11:24:36.886673 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":312,"threads_started":1}
I20260812 11:24:36.887797 25900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:37.013171 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 25710
I20260812 11:24:37.045107 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 25921
Aug 12 11:24:37 dist-test-slave-h6n0 krb5kdc[25286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.12.110.195@KRBTEST.COM
I20260812 11:24:37.064983 26375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:37.046263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55512 (local address 127.12.110.195:41305)
0812 11:24:37.047115 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:37.047120 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:37.047136 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:24:37.047195 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:37.047199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:37.047263 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:37.047358 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:37.049432 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:37.050746 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:37.052261 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:37.052694 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:37.052798 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:37.057565 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:37.057584 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:37.057602 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:37.057652 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:37.061203 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:37.061905 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:37.061913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:37.061915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:37.062470 (+ 555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:37.063264 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:37.063269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:37.063271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:37.063679 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:37.063924 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:37.064348 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 11:24:37.064606 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":596,"threads_started":1}
I20260812 11:24:37.065718 26150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:37.217082 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 25921
I20260812 11:24:37.257477 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 25306
I20260812 11:24:37.258862 25425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:37.379966 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 25306
2026-08-12T11:24:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6514 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:24:37 dist-test-slave-h6n0 krb5kdc[26387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:24:37 dist-test-slave-h6n0 krb5kdc[26387](info): set up 2 sockets
Aug 12 11:24:37 dist-test-slave-h6n0 krb5kdc[26387](info): commencing operation
krb5kdc: starting...
W20260812 11:24:38.808162 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.368s user 0.004s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:24:39 dist-test-slave-h6n0 krb5kdc[26387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:24:39.134847 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:45677
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:45677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:39.482501 26411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:39.482928 26411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:39.483107 26411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:39.490453 26411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:39.490629 26411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:39.490742 26411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:39.490850 26411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:39.490932 26411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:39.504338 26411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:45677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:45677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:39.505834 26411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:39.507238 26411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:39.519408 26423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:39.519455 26429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:39.519438 26426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:39.520195 26411 server_base.cc:1034] running on GCE node
I20260812 11:24:39.521014 26411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:39.522454 26411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:39.523705 26411 hybrid_clock.cc:648] HybridClock initialized: now 1786533879523645 us; error 50 us; skew 500 ppm
I20260812 11:24:39.524051 26411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-12T11:24:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2209 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:24:39 dist-test-slave-h6n0 krb5kdc[26441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:24:39 dist-test-slave-h6n0 krb5kdc[26441](info): set up 2 sockets
Aug 12 11:24:39 dist-test-slave-h6n0 krb5kdc[26441](info): commencing operation
krb5kdc: starting...
W20260812 11:24:41.025751 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.375s user 0.002s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:24:41 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:24:41.322769 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37905
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:41.663194 26461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:41.663545 26461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:41.663712 26461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:41.670812 26461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:41.670957 26461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:41.671068 26461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:41.671192 26461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:41.671280 26461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:41.681874 26461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:41.683280 26461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:41.684602 26461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:41.696615 26473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:41.696611 26475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:41.696945 26481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:41.697768 26461 server_base.cc:1034] running on GCE node
I20260812 11:24:41.698706 26461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:41.700138 26461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:41.701385 26461 hybrid_clock.cc:648] HybridClock initialized: now 1786533881701352 us; error 47 us; skew 500 ppm
I20260812 11:24:41.701747 26461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:41 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533881, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:41.704970 26461 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:24:41.706614 26461 webserver.cc:460] Webserver started at http://127.12.110.254:40143/ using document root <none> and password file <none>
I20260812 11:24:41.707294 26461 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:41.707441 26461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:41.707764 26461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:41.710762 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "bbad92031a34448fa534f81abf027748"
format_stamp: "Formatted at 2026-08-12 11:24:41 on dist-test-slave-h6n0"
server_key: "0f13c2a36f3b4059b69b036329a650f7"
server_key_iv: "cece6e8a28be9458705e3994a0be2f02"
server_key_version: "encryptionkey@0"
I20260812 11:24:41.711508 26461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "bbad92031a34448fa534f81abf027748"
format_stamp: "Formatted at 2026-08-12 11:24:41 on dist-test-slave-h6n0"
server_key: "0f13c2a36f3b4059b69b036329a650f7"
server_key_iv: "cece6e8a28be9458705e3994a0be2f02"
server_key_version: "encryptionkey@0"
I20260812 11:24:41.716471 26461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:24:41.719911 26493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:41.721210 26461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:24:41.721414 26461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "bbad92031a34448fa534f81abf027748"
format_stamp: "Formatted at 2026-08-12 11:24:41 on dist-test-slave-h6n0"
server_key: "0f13c2a36f3b4059b69b036329a650f7"
server_key_iv: "cece6e8a28be9458705e3994a0be2f02"
server_key_version: "encryptionkey@0"
I20260812 11:24:41.721614 26461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:41.747521 26461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:41.751865 26461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:41.752156 26461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:41.770928 26461 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37905
I20260812 11:24:41.770977 26570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37905 every 8 connection(s)
I20260812 11:24:41.772274 26461 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:41.773105 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 26461
I20260812 11:24:41.773365 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:24:41.774235 12731 external_mini_cluster.cc:1428] Setting key 2539e88945116a739cb12949038c7add
I20260812 11:24:41.777132 26571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 11:24:41 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:41.790287 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Bootstrap starting.
I20260812 11:24:41.794400 26571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Neither blocks nor log segments found. Creating new log.
I20260812 11:24:41.795890 26571 log.cc:824] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:41.799399 26571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: No bootstrap required, opened a new log
I20260812 11:24:41.800567 26576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:41.776361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51638 (local address 127.12.110.254:37905)
0812 11:24:41.777146 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:41.777159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:41.777227 (+ 68us) server_negotiation.cc:407] Connection header received
0812 11:24:41.778961 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:41.778981 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:41.779309 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:41.779649 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:41.781506 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:41.782914 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:41.784680 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:41.785382 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:41.785548 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:41.790386 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:41.790395 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:41.790435 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:41.790495 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:41.795894 (+ 5399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:41.797073 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:41.797084 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:41.797095 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:41.797342 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:41.797835 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:41.797839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:41.797841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:41.798367 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:41.798683 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:41.799138 (+ 455us) server_negotiation.cc:299] Negotiation successful
0812 11:24:41.799517 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":374,"threads_started":1}
I20260812 11:24:41.807152 26571 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:41.807546 26571 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:24:41.807658 26571 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbad92031a34448fa534f81abf027748, State: Initialized, Role: FOLLOWER
I20260812 11:24:41.808211 26571 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:41.808459 26571 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:41.808611 26571 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:41.808789 26571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:24:41.811304 26571 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:41.811794 26571 leader_election.cc:302] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbad92031a34448fa534f81abf027748; no voters:
I20260812 11:24:41.812407 26571 leader_election.cc:288] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:24:41.812551 26580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:24:41.813355 26580 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 LEADER]: Becoming Leader. State: Replica: bbad92031a34448fa534f81abf027748, State: Running, Role: LEADER
I20260812 11:24:41.813910 26580 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:41.814615 26571 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:41.816684 26584 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbad92031a34448fa534f81abf027748. Latest consensus state: current_term: 1 leader_uuid: "bbad92031a34448fa534f81abf027748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } } }
I20260812 11:24:41.817003 26584 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:41.822388 26582 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbad92031a34448fa534f81abf027748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } } }
I20260812 11:24:41.823467 26582 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:41.823959 26592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:41.828290 26592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:41.836331 26592 catalog_manager.cc:1347] Generated new cluster ID: 01f66910a19f420e9d9a23b7ecc905ba
I20260812 11:24:41.836439 26592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:41.850347 26592 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:24:41.851590 26592 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:41.864493 26592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Generated new TSK 0
I20260812 11:24:41.865342 26592 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:24:41.940032 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37905
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:42.292979 26619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:42.293267 26619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:42.293437 26619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:42.301949 26619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:42.302068 26619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:42.302249 26619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:42.312422 26619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:42.313762 26619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:42.315264 26619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:42.329316 26634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:42.329255 26632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:42.329221 26638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:42.330423 26619 server_base.cc:1034] running on GCE node
I20260812 11:24:42.331172 26619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:42.332145 26619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:42.333400 26619 hybrid_clock.cc:648] HybridClock initialized: now 1786533882333342 us; error 71 us; skew 500 ppm
I20260812 11:24:42.333742 26619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:42 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:42.337337 26619 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:24:42.339010 26619 webserver.cc:460] Webserver started at http://127.12.110.193:42013/ using document root <none> and password file <none>
I20260812 11:24:42.339725 26619 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:42.339881 26619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:42.340204 26619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:42.343194 26619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "a4c1007591d64e9e821c75fa6da2f55c"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "05582752a444c97d9b334e30351d0bf9"
server_key_iv: "e5d680f47dc9f4e0d32f59638399814e"
server_key_version: "encryptionkey@0"
I20260812 11:24:42.344009 26619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "a4c1007591d64e9e821c75fa6da2f55c"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "05582752a444c97d9b334e30351d0bf9"
server_key_iv: "e5d680f47dc9f4e0d32f59638399814e"
server_key_version: "encryptionkey@0"
I20260812 11:24:42.349309 26619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:24:42.352772 26647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:42.354090 26619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 11:24:42.354288 26619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "a4c1007591d64e9e821c75fa6da2f55c"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "05582752a444c97d9b334e30351d0bf9"
server_key_iv: "e5d680f47dc9f4e0d32f59638399814e"
server_key_version: "encryptionkey@0"
I20260812 11:24:42.354497 26619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:42.386325 26619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:42.391026 26619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:42.391346 26619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:42.392371 26619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:42.393850 26619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:42.393960 26619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:42.394125 26619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:42.394212 26619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:42.433144 26619 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:39827
I20260812 11:24:42.433199 26832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:39827 every 8 connection(s)
I20260812 11:24:42.434505 26619 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:42.440043 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 26619
I20260812 11:24:42.440270 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:24:42.441150 12731 external_mini_cluster.cc:1428] Setting key 2f720d788e6ee357b119641a1f3721d3
Aug 12 11:24:42 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:42.456923 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:42.437196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:42131 (local address 127.12.110.254:37905)
0812 11:24:42.437660 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:42.437665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:42.438719 (+ 1054us) server_negotiation.cc:407] Connection header received
0812 11:24:42.440647 (+ 1928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:42.440653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:42.440748 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:42.440887 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:42.442627 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:42.443630 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:42.445241 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:42.445997 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:42.446150 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:42.449615 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:42.449623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:42.449631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:42.449674 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:42.453309 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:42.454165 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:42.454172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:42.454178 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:42.454314 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:42.454823 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:42.454830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:42.454844 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:42.455177 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:42.455322 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:42.456216 (+ 894us) server_negotiation.cc:299] Negotiation successful
0812 11:24:42.456458 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":246,"threads_started":1}
I20260812 11:24:42.457654 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:42.437764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:42131)
0812 11:24:42.438483 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:42.438591 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:42.440365 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:42.441088 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:42.441096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:42.441657 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:42.442426 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:42.442450 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:42.443813 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:42.443829 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:42.445077 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:42.445089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:42.446230 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:42.446235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:42.446440 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:42.447187 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:42.447209 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:42.449384 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:42.453458 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:42.453468 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:42.453484 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:42.454028 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:42.454428 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:42.454435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:42.454440 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:42.454691 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:42.455338 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:42.455347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:42.455779 (+ 432us) client_negotiation.cc:769] Sending connection context
0812 11:24:42.456096 (+ 317us) client_negotiation.cc:240] Negotiation successful
0812 11:24:42.456591 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":305,"threads_started":1}
I20260812 11:24:42.459231 26833 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37905
I20260812 11:24:42.459543 26833 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:42.460376 26833 heartbeater.cc:511] Master 127.12.110.254:37905 requested a full tablet report, sending...
I20260812 11:24:42.462618 26514 ts_manager.cc:194] Registered new tserver with Master: a4c1007591d64e9e821c75fa6da2f55c (127.12.110.193:39827)
I20260812 11:24:42.464113 26514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:42131
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:24:42.515866 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37905
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:42.900075 26842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:42.900417 26842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:42.900605 26842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:42.907779 26842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:42.907967 26842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:42.908128 26842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:42.917699 26842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:42.918958 26842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:42.920449 26842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:42.934235 26849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:42.934298 26851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:42.934322 26857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:42.935468 26842 server_base.cc:1034] running on GCE node
I20260812 11:24:42.936328 26842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:42.937440 26842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:42.938859 26842 hybrid_clock.cc:648] HybridClock initialized: now 1786533882938773 us; error 91 us; skew 500 ppm
I20260812 11:24:42.939368 26842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:42 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:42.943252 26842 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:24:42.945034 26842 webserver.cc:460] Webserver started at http://127.12.110.194:36957/ using document root <none> and password file <none>
I20260812 11:24:42.945765 26842 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:42.945876 26842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:42.946242 26842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:42.949292 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "271d9105de044f47a1875bdfb23d21c7"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "f7ba2e3ab51f8001a76926331b784568"
server_key_iv: "612a073d123f1ddd2bb4c55de23f98c0"
server_key_version: "encryptionkey@0"
I20260812 11:24:42.950098 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "271d9105de044f47a1875bdfb23d21c7"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "f7ba2e3ab51f8001a76926331b784568"
server_key_iv: "612a073d123f1ddd2bb4c55de23f98c0"
server_key_version: "encryptionkey@0"
I20260812 11:24:42.956246 26842 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 11:24:42.959995 26867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:42.961328 26842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:24:42.961527 26842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "271d9105de044f47a1875bdfb23d21c7"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "f7ba2e3ab51f8001a76926331b784568"
server_key_iv: "612a073d123f1ddd2bb4c55de23f98c0"
server_key_version: "encryptionkey@0"
I20260812 11:24:42.961746 26842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:42.983649 26842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:42.989102 26842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:42.989423 26842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:42.990463 26842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:42.991876 26842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:42.992002 26842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:42.992162 26842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:42.992238 26842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:43.029270 26842 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:33639
I20260812 11:24:43.029327 27073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:33639 every 8 connection(s)
I20260812 11:24:43.030619 26842 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:43.034323 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 26842
I20260812 11:24:43.034581 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:24:43.035133 12731 external_mini_cluster.cc:1428] Setting key dd9004109f35aa2b8d430c1931526f42
Aug 12 11:24:43 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533882, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:43.054549 27076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:43.033605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:35761 (local address 127.12.110.254:37905)
0812 11:24:43.034445 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:43.034455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:43.035409 (+ 954us) server_negotiation.cc:407] Connection header received
0812 11:24:43.037732 (+ 2323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:43.037737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:43.037842 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:43.037966 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:43.039605 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.040368 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.041925 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.042380 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.042518 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:43.046627 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:43.046633 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:43.046636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:43.046677 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:43.051008 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.052004 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.052008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.052011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.052174 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.052642 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.052646 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.052648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.052936 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:43.053116 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:43.053828 (+ 712us) server_negotiation.cc:299] Negotiation successful
0812 11:24:43.054146 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":410,"threads_started":1}
I20260812 11:24:43.055543 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:43.033916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:35761)
0812 11:24:43.035108 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:43.035221 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:43.037440 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:43.038220 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:43.038228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:43.038861 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:43.039427 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:43.039445 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.040540 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.040544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:43.041772 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:43.041783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.042660 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.042668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:43.042898 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:43.043646 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:43.043676 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:43.046374 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:43.051322 (+ 4948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:43.051330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:43.051343 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:43.051822 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:43.052296 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:43.052301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:43.052304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:43.052507 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:43.053082 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:43.053091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:43.053440 (+ 349us) client_negotiation.cc:769] Sending connection context
0812 11:24:43.053728 (+ 288us) client_negotiation.cc:240] Negotiation successful
0812 11:24:43.054284 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"thread_start_us":473,"threads_started":1}
I20260812 11:24:43.057885 27074 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37905
I20260812 11:24:43.058262 27074 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:43.059585 27074 heartbeater.cc:511] Master 127.12.110.254:37905 requested a full tablet report, sending...
I20260812 11:24:43.061576 26514 ts_manager.cc:194] Registered new tserver with Master: 271d9105de044f47a1875bdfb23d21c7 (127.12.110.194:33639)
I20260812 11:24:43.062427 26514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:35761
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:24:43.109424 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37905
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:24:43.467921 26833 heartbeater.cc:503] Master 127.12.110.254:37905 was elected leader, sending a full tablet report...
W20260812 11:24:43.475355 27083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:43.475672 27083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:43.475848 27083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:43.485193 27083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:43.485383 27083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:43.485599 27083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:43.496037 27083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:43.497313 27083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:43.498669 27083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:43.511746 27090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:43.511979 27091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:43.512090 27093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:43.512735 27083 server_base.cc:1034] running on GCE node
I20260812 11:24:43.513386 27083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:43.514273 27083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:43.515511 27083 hybrid_clock.cc:648] HybridClock initialized: now 1786533883515422 us; error 53 us; skew 500 ppm
I20260812 11:24:43.515811 27083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:43 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:43.519243 27083 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:24:43.520828 27083 webserver.cc:460] Webserver started at http://127.12.110.195:44445/ using document root <none> and password file <none>
I20260812 11:24:43.521771 27083 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:43.521912 27083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:43.522230 27083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:24:43.525254 27083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "6a12b82a23a8498588c73a160e13f00e"
format_stamp: "Formatted at 2026-08-12 11:24:43 on dist-test-slave-h6n0"
server_key: "9d0b309900078ce2abf97bd258613ea7"
server_key_iv: "647d21a03b1590533913288681758769"
server_key_version: "encryptionkey@0"
I20260812 11:24:43.526253 27083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "6a12b82a23a8498588c73a160e13f00e"
format_stamp: "Formatted at 2026-08-12 11:24:43 on dist-test-slave-h6n0"
server_key: "9d0b309900078ce2abf97bd258613ea7"
server_key_iv: "647d21a03b1590533913288681758769"
server_key_version: "encryptionkey@0"
I20260812 11:24:43.531634 27083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:24:43.535017 27108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:43.536227 27083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:24:43.536417 27083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "6a12b82a23a8498588c73a160e13f00e"
format_stamp: "Formatted at 2026-08-12 11:24:43 on dist-test-slave-h6n0"
server_key: "9d0b309900078ce2abf97bd258613ea7"
server_key_iv: "647d21a03b1590533913288681758769"
server_key_version: "encryptionkey@0"
I20260812 11:24:43.536615 27083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:43.554308 27083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:43.559669 27083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:43.559988 27083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:43.561064 27083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:43.562492 27083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:43.562662 27083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 11:24:43.562824 27083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:43.562899 27083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:43.599604 27083 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:41925
I20260812 11:24:43.599699 27309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:41925 every 8 connection(s)
I20260812 11:24:43.600931 27083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:43.602033 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 27083
I20260812 11:24:43.602218 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:24:43.602808 12731 external_mini_cluster.cc:1428] Setting key b7211ab32a2da6c881d351f8724b148d
Aug 12 11:24:43 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533883, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:24:43.621748 27318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:43.603865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:41665 (local address 127.12.110.254:37905)
0812 11:24:43.604428 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:43.604433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:43.605538 (+ 1105us) server_negotiation.cc:407] Connection header received
0812 11:24:43.607586 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:43.607590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:43.607666 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:43.607758 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:43.609169 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.609990 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.611552 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.611943 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.612039 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:43.615667 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:43.615671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:43.615674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:43.615705 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:43.618610 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.619637 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.619641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.619643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.619766 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.620331 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.620334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.620336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.620598 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:43.620717 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:43.621317 (+ 600us) server_negotiation.cc:299] Negotiation successful
0812 11:24:43.621472 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":250,"threads_started":1}
I20260812 11:24:43.622774 27321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:43.604492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:41665)
0812 11:24:43.605349 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:43.605410 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:43.607351 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:43.607942 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:43.607950 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:43.608457 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:43.608987 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:43.609003 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.610137 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.610141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:43.611416 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:43.611429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.612168 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.612174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:43.612367 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:43.613045 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:43.613072 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:43.615480 (+ 2408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:24:43.618752 (+ 3272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:43.618762 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:43.618778 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:43.619501 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:43.619900 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:24:43.619903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:24:43.619906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:24:43.620171 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:24:43.620707 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:24:43.620713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:24:43.621000 (+ 287us) client_negotiation.cc:769] Sending connection context
0812 11:24:43.621221 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 11:24:43.621583 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":350,"threads_started":1}
I20260812 11:24:43.624187 27314 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37905
I20260812 11:24:43.624567 27314 heartbeater.cc:464] Registering TS with master...
I20260812 11:24:43.625327 27314 heartbeater.cc:511] Master 127.12.110.254:37905 requested a full tablet report, sending...
I20260812 11:24:43.627074 26514 ts_manager.cc:194] Registered new tserver with Master: 6a12b82a23a8498588c73a160e13f00e (127.12.110.195:41925)
I20260812 11:24:43.627864 26514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:41665
I20260812 11:24:43.632130 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab.
Entry for principal oryx/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab.
WARNING: no policy specified for oryx/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab.
Entry for principal oryx/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab.
WARNING: no policy specified for oryx/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab.
Entry for principal oryx/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab.
I20260812 11:24:43.830410 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 26619
Aug 12 11:24:43 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:24:43.850521 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:43.832021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.12.110.193:39827)
0812 11:24:43.832860 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:43.832868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:43.832897 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:24:43.832960 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:43.832967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:43.833185 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:43.833320 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:43.835447 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.836643 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.838720 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.839697 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.839854 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:43.844779 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:43.844794 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:43.844811 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:43.844850 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:43.847941 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.848675 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.848681 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.848684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.848816 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.849138 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.849142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.849143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.849504 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:43.849748 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:43.850003 (+ 255us) server_negotiation.cc:299] Negotiation successful
0812 11:24:43.850220 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":509,"threads_started":1}
I20260812 11:24:43.852087 26825 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:43.939188 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 26619
I20260812 11:24:43.964388 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 26842
Aug 12 11:24:43 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:24:43.982275 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:43.965265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.12.110.194:33639)
0812 11:24:43.965871 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:43.965878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:43.965903 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:24:43.965966 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:43.965973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:43.966225 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:43.966363 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:43.968667 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.969685 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.971036 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:43.971593 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:43.971690 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:43.975978 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:43.975989 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:43.976002 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:43.976045 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:43.979405 (+ 3360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.980244 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.980251 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.980254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.980409 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:43.980831 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:43.980844 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:43.980846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:43.981203 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:43.981461 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:43.981761 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 11:24:43.981963 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":313,"threads_started":1}
I20260812 11:24:43.983599 27062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:44.065651 27074 heartbeater.cc:503] Master 127.12.110.254:37905 was elected leader, sending a full tablet report...
I20260812 11:24:44.067898 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 26842
I20260812 11:24:44.096093 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 27083
Aug 12 11:24:44 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:24:44.112977 27343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:44.097230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.12.110.195:41925)
0812 11:24:44.097758 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:44.097766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:44.097796 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:24:44.097872 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:44.097878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:44.098083 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:44.098262 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:44.099664 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:44.100649 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:44.101918 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:44.102471 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:44.102735 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:44.107260 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:44.107271 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:44.107285 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:44.107318 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:44.110299 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:44.111079 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:44.111086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:44.111088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:44.111208 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:44.111554 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:44.111557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:44.111559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:44.111895 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:44.112132 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:44.112380 (+ 248us) server_negotiation.cc:299] Negotiation successful
0812 11:24:44.112605 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":245,"threads_started":1}
I20260812 11:24:44.114251 27303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:44.198319 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 27083
I20260812 11:24:44.219609 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 26461
I20260812 11:24:44.221057 26565 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:24:44.332279 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 26461
I20260812 11:24:44.362097 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37905
--webserver_interface=127.12.110.254
--webserver_port=40143
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:24:44.705538 27348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:44.705860 27348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:44.706053 27348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:44.713973 27348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:24:44.714087 27348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:44.714192 27348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:44.714284 27348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:24:44.714358 27348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:24:44.724076 27348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=40143
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:44.725396 27348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:44.726734 27348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:44.739137 27362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:44.739369 27358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:44.739621 27357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:44.740026 27348 server_base.cc:1034] running on GCE node
I20260812 11:24:44.740849 27348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:44.742244 27348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:44.743490 27348 hybrid_clock.cc:648] HybridClock initialized: now 1786533884743427 us; error 89 us; skew 500 ppm
I20260812 11:24:44.743871 27348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:44 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533884, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:44.751950 27348 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:24:44.753477 27348 webserver.cc:460] Webserver started at http://127.12.110.254:40143/ using document root <none> and password file <none>
I20260812 11:24:44.754160 27348 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:44.754323 27348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:44.759745 27348 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:24:44.762634 27371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:44.764822 27348 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 11:24:44.765156 27348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "bbad92031a34448fa534f81abf027748"
format_stamp: "Formatted at 2026-08-12 11:24:41 on dist-test-slave-h6n0"
server_key: "0f13c2a36f3b4059b69b036329a650f7"
server_key_iv: "cece6e8a28be9458705e3994a0be2f02"
server_key_version: "encryptionkey@0"
I20260812 11:24:44.766812 27348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:44.787459 27348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:44.791973 27348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:44.792260 27348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:44.811655 27348 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37905
I20260812 11:24:44.811664 27480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37905 every 8 connection(s)
I20260812 11:24:44.813194 27348 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:24:44.819869 27481 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:24:44.820847 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 27348
I20260812 11:24:44.821841 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:39827
--local_ip_for_outbound_sockets=127.12.110.193
--tserver_master_addrs=127.12.110.254:37905
--webserver_port=42013
--webserver_interface=127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:24:44.824579 27481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Bootstrap starting.
I20260812 11:24:44.846414 27481 log.cc:824] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Log is configured to *not* fsync() on all Append() calls
I20260812 11:24:44.855062 27481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:24:44.855652 27481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Bootstrap complete.
I20260812 11:24:44.864738 27481 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:44.865571 27481 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbad92031a34448fa534f81abf027748, State: Initialized, Role: FOLLOWER
I20260812 11:24:44.866197 27481 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:44.866451 27481 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:24:44.866771 27481 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:24:44.866991 27481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:24:44.870891 27481 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:44.871377 27481 leader_election.cc:302] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbad92031a34448fa534f81abf027748; no voters:
I20260812 11:24:44.872032 27481 leader_election.cc:288] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:24:44.872246 27490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:24:44.873421 27490 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [term 2 LEADER]: Becoming Leader. State: Replica: bbad92031a34448fa534f81abf027748, State: Running, Role: LEADER
I20260812 11:24:44.874086 27481 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:24:44.874150 27490 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } }
I20260812 11:24:44.880424 27491 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bbad92031a34448fa534f81abf027748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } } }
I20260812 11:24:44.880874 27491 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:44.881476 27493 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbad92031a34448fa534f81abf027748. Latest consensus state: current_term: 2 leader_uuid: "bbad92031a34448fa534f81abf027748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbad92031a34448fa534f81abf027748" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37905 } } }
I20260812 11:24:44.881820 27493 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748 [sys.catalog]: This master's current role is: LEADER
I20260812 11:24:44.902885 27507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:24:44.910202 27507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:24:44.912111 27507 catalog_manager.cc:1259] Loaded cluster ID: 01f66910a19f420e9d9a23b7ecc905ba
I20260812 11:24:44.912293 27507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:24:44.917483 27507 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:24:44.919401 27507 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bbad92031a34448fa534f81abf027748: Loaded TSK: 0
I20260812 11:24:44.921299 27507 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:24:44.925660 27518 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260812 11:24:45.211844 27486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:45.212551 27486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:45.212888 27486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:45.220198 27486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:45.220369 27486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:45.220556 27486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:24:45.230333 27486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.193.keytab
--principal=oryx/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:39827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=42013
--tserver_master_addrs=127.12.110.254:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:45.231763 27486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:45.233057 27486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:45.247046 27526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:45.246978 27527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:45.247937 27532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:45.249330 27486 server_base.cc:1034] running on GCE node
I20260812 11:24:45.249992 27486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:45.251046 27486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:45.252270 27486 hybrid_clock.cc:648] HybridClock initialized: now 1786533885252216 us; error 45 us; skew 500 ppm
I20260812 11:24:45.252588 27486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533885, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:45.256107 27486 init.cc:377] Logged in from keytab as oryx/127.12.110.193@KRBTEST.COM (short username oryx)
I20260812 11:24:45.257649 27486 webserver.cc:460] Webserver started at http://127.12.110.193:42013/ using document root <none> and password file <none>
I20260812 11:24:45.258354 27486 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:45.258517 27486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:45.263942 27486 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260812 11:24:45.266644 27543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:45.268242 27486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:24:45.268456 27486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "a4c1007591d64e9e821c75fa6da2f55c"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "05582752a444c97d9b334e30351d0bf9"
server_key_iv: "e5d680f47dc9f4e0d32f59638399814e"
server_key_version: "encryptionkey@0"
I20260812 11:24:45.269459 27486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:45.312431 27486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:45.317461 27486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:45.317770 27486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:45.319146 27486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:45.321452 27486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:45.321611 27486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:45.321765 27486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:45.321861 27486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:45.358042 27486 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:39827
I20260812 11:24:45.358130 27763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:39827 every 8 connection(s)
I20260812 11:24:45.359543 27486 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:24:45.364168 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 27486
I20260812 11:24:45.364807 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:33639
--local_ip_for_outbound_sockets=127.12.110.194
--tserver_master_addrs=127.12.110.254:37905
--webserver_port=36957
--webserver_interface=127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:45.378734 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:45.362747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:54291)
0812 11:24:45.363627 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:45.363666 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:45.365509 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:45.366834 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:45.366847 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:45.367459 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:45.367971 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.367988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.369558 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.369563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.370977 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.370990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.372053 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.372061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.372325 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.373070 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:45.373092 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:45.377469 (+ 4377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":362,"threads_started":1}
W20260812 11:24:45.378823 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:45.362674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:54291 (local address 127.12.110.254:37905)
0812 11:24:45.363481 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:45.363495 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:45.363786 (+ 291us) server_negotiation.cc:407] Connection header received
0812 11:24:45.366026 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:45.366038 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:45.366325 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:45.366695 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:45.368205 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.369395 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.371125 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.371806 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.372021 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.377805 (+ 5784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:54291: BlockingRecv error: recv got EOF from 127.12.110.193:54291 (error 108)
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":287,"threads_started":1}
W20260812 11:24:45.379513 27764 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:37905 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.110.254:37905: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:45.388115 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:45.380128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:44497)
0812 11:24:45.380294 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:45.380311 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:45.380425 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:45.380783 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:45.380788 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:45.381135 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:45.381263 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.381271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.382265 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.382270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.383237 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.383249 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.383914 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.383920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.384100 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.384738 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:45.384755 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:45.387817 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:24:45.388254 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:45.380329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:44497 (local address 127.12.110.254:37905)
0812 11:24:45.380544 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:45.380549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:45.380562 (+ 13us) server_negotiation.cc:407] Connection header received
0812 11:24:45.380602 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:45.380605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:45.380657 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:45.380759 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:45.381402 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.382124 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.383374 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.383772 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.383902 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.388009 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:44497: BlockingRecv error: recv got EOF from 127.12.110.193:44497 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:45.398025 27767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:45.389062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:56697)
0812 11:24:45.389223 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:45.389242 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:45.389366 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:45.389776 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:45.389782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:45.390118 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:45.390269 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.390278 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.391528 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.391534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.392578 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.392590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.393306 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.393311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.393508 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.394215 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:45.394232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:45.397692 (+ 3460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:24:45.398290 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:45.389273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:56697 (local address 127.12.110.254:37905)
0812 11:24:45.389478 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:45.389484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:45.389508 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:24:45.389549 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:45.389552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:45.389627 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:45.389750 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:45.390423 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.391357 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.392738 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.393152 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.393284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.397896 (+ 4612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:56697: BlockingRecv error: recv got EOF from 127.12.110.193:56697 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260812 11:24:45.398494 27764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:24:45.713445 27771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:45.713721 27771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:45.713817 27771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:45.720945 27771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:45.721057 27771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:45.721171 27771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:24:45.730623 27771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.194.keytab
--principal=oryx/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:33639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=36957
--tserver_master_addrs=127.12.110.254:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:45.731813 27771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:45.733016 27771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:45.745919 27780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:45.746054 27786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:45.746090 27781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:45.746717 27771 server_base.cc:1034] running on GCE node
I20260812 11:24:45.747237 27771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:45.748075 27771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:45.749320 27771 hybrid_clock.cc:648] HybridClock initialized: now 1786533885749276 us; error 35 us; skew 500 ppm
I20260812 11:24:45.749703 27771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533885, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:45.753646 27771 init.cc:377] Logged in from keytab as oryx/127.12.110.194@KRBTEST.COM (short username oryx)
I20260812 11:24:45.755373 27771 webserver.cc:460] Webserver started at http://127.12.110.194:36957/ using document root <none> and password file <none>
I20260812 11:24:45.756151 27771 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:45.756318 27771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:45.761639 27771 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:24:45.764312 27800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:45.765622 27771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:24:45.765821 27771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "271d9105de044f47a1875bdfb23d21c7"
format_stamp: "Formatted at 2026-08-12 11:24:42 on dist-test-slave-h6n0"
server_key: "f7ba2e3ab51f8001a76926331b784568"
server_key_iv: "612a073d123f1ddd2bb4c55de23f98c0"
server_key_version: "encryptionkey@0"
I20260812 11:24:45.766916 27771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:45.786752 27771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:45.790994 27771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:45.791297 27771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:45.792516 27771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:45.794191 27771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:45.794337 27771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:45.794508 27771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:45.794646 27771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260812 11:24:45.833159 27771 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:33639
I20260812 11:24:45.833249 28000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:33639 every 8 connection(s)
I20260812 11:24:45.834601 27771 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:24:45.837666 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 27771
I20260812 11:24:45.838452 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:41925
--local_ip_for_outbound_sockets=127.12.110.195
--tserver_master_addrs=127.12.110.254:37905
--webserver_port=44445
--webserver_interface=127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:24:45.851397 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:45.838067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:49051 (local address 127.12.110.254:37905)
0812 11:24:45.838295 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:45.838301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:45.839629 (+ 1328us) server_negotiation.cc:407] Connection header received
0812 11:24:45.841470 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:45.841476 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:45.841570 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:45.841693 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:45.843306 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.844033 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.845549 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.845985 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.846122 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.851203 (+ 5081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:49051: BlockingRecv error: recv got EOF from 127.12.110.194:49051 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:24:45.852447 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:45.838602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:49051)
0812 11:24:45.839436 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:45.839483 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:45.841201 (+ 1718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:45.841927 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:45.841934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:45.842435 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:45.843127 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.843144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.844211 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.844215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.845386 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.845398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.846220 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.846228 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.846463 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.847231 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:45.847257 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:45.851077 (+ 3820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":360,"threads_started":1}
W20260812 11:24:45.853255 28002 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:37905 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.110.254:37905: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:45.863390 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:45.853843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:32871)
0812 11:24:45.854050 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:45.854071 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:45.854174 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:45.854473 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:45.854481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:45.854945 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:45.855194 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.855211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.856241 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.856248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.857904 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.857922 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.858843 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.858851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.859116 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.859725 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:45.859752 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:45.863074 (+ 3322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260812 11:24:45.864130 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:45.853999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:32871 (local address 127.12.110.254:37905)
0812 11:24:45.854181 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:45.854185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:45.854202 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:24:45.854270 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:45.854274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:45.854331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:45.854410 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:45.855357 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.856091 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.858065 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.858636 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.858767 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.863390 (+ 4623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:32871: BlockingRecv error: recv got EOF from 127.12.110.194:32871 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:45 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:45.873363 28004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:45.864367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:41733)
0812 11:24:45.864675 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:45.864693 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:45.864791 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:45.865230 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:45.865233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:45.865494 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:45.865628 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.865635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.867200 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.867205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.868341 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:45.868352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.869392 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.869397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:45.869554 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.870138 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:45.870155 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:45.873149 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189}
W20260812 11:24:45.873740 28002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:24:45.874001 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:45.864717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:41733 (local address 127.12.110.254:37905)
0812 11:24:45.864915 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:45.864922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:45.864943 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:24:45.864997 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:45.865005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:45.865088 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:45.865200 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:45.865770 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.867021 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.868500 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:45.869242 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:45.869390 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:45.873411 (+ 4021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:41733: BlockingRecv error: recv got EOF from 127.12.110.194:41733 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260812 11:24:46.184969 28006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:24:46.185266 28006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:24:46.185380 28006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:24:46.192721 28006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:24:46.192839 28006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:24:46.192960 28006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:24:46.203104 28006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/krb5kdc/oryx_127.12.110.195.keytab
--principal=oryx/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:41925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=44445
--tserver_master_addrs=127.12.110.254:37905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:24:46.204360 28006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:24:46.205595 28006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:24:46.218725 28018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:46.218792 28015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:24:46.218816 28022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:24:46.220185 28006 server_base.cc:1034] running on GCE node
I20260812 11:24:46.220799 28006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:24:46.222343 28006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:24:46.223675 28006 hybrid_clock.cc:648] HybridClock initialized: now 1786533886223603 us; error 60 us; skew 500 ppm
I20260812 11:24:46.224071 28006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533886, etypes {rep=17 tkt=17 ses=17}, oryx/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:24:46.227481 28006 init.cc:377] Logged in from keytab as oryx/127.12.110.195@KRBTEST.COM (short username oryx)
I20260812 11:24:46.229260 28006 webserver.cc:460] Webserver started at http://127.12.110.195:44445/ using document root <none> and password file <none>
I20260812 11:24:46.229969 28006 fs_manager.cc:359] Metadata directory not provided
I20260812 11:24:46.230140 28006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:24:46.236620 28006 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260812 11:24:46.240861 28039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:46.242837 28006 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:24:46.243072 28006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "6a12b82a23a8498588c73a160e13f00e"
format_stamp: "Formatted at 2026-08-12 11:24:43 on dist-test-slave-h6n0"
server_key: "9d0b309900078ce2abf97bd258613ea7"
server_key_iv: "647d21a03b1590533913288681758769"
server_key_version: "encryptionkey@0"
I20260812 11:24:46.244195 28006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:24:46.268159 28006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:24:46.273411 28006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:24:46.273908 28006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:24:46.275913 28006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:24:46.277817 28006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:24:46.277946 28006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:46.278081 28006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:24:46.278167 28006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:24:46.315526 28006 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:41925
I20260812 11:24:46.315644 28246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:41925 every 8 connection(s)
I20260812 11:24:46.317022 28006 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:24:46.323990 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 28006
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
W20260812 11:24:46.336668 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:46.320718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:49103 (local address 127.12.110.254:37905)
0812 11:24:46.320946 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:46.320951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:46.322338 (+ 1387us) server_negotiation.cc:407] Connection header received
0812 11:24:46.324294 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:46.324300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:46.324391 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:46.324512 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:46.326623 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.327390 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.329251 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.329645 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.329734 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.336270 (+ 6536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:49103: BlockingRecv error: recv got EOF from 127.12.110.195:49103 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 11:24:46.337345 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:46.321341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:49103)
0812 11:24:46.322156 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:46.322197 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:46.324018 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:46.324771 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:46.324785 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:46.325572 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:46.326366 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.326396 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.327562 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.327569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.329116 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.329130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.329878 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.329887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.330199 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.330971 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:46.330999 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:46.336025 (+ 5026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":325,"threads_started":1}
W20260812 11:24:46.338024 28251 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:37905 (0 consecutive failures): Not authorized: Failed to ping master at 127.12.110.254:37905: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
I20260812 11:24:46.342180 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:46.325225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.12.110.254:37905)
0812 11:24:46.325870 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:46.325875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:46.325892 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:24:46.325940 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:46.325943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:46.325998 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:46.326093 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:46.327848 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.328680 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.330284 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.330793 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.330940 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.334913 (+ 3973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:24:46.334922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:24:46.334938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:24:46.334983 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:24:46.339055 (+ 4072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:46.339739 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:46.339750 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:46.339763 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:46.339931 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:24:46.340312 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:24:46.340317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:24:46.340320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:24:46.340772 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:24:46.341122 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0812 11:24:46.341563 (+ 441us) server_negotiation.cc:299] Negotiation successful
0812 11:24:46.341913 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":404,"threads_started":1}
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:46.347118 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:46.338687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:32913)
0812 11:24:46.339013 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:46.339028 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:46.339119 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:46.339440 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:46.339444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:46.339780 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:46.339907 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.339915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.341122 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.341128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.342127 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.342139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.343115 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.343119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.343243 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.343744 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:46.343766 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:46.346937 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260812 11:24:46.347293 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:46.338809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:32913 (local address 127.12.110.254:37905)
0812 11:24:46.338962 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:46.338966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:46.339106 (+ 140us) server_negotiation.cc:407] Connection header received
0812 11:24:46.339238 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:46.339243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:46.339312 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:46.339411 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:46.340059 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.340983 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.342279 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.342956 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.343071 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.347100 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:32913: BlockingRecv error: recv got EOF from 127.12.110.195:32913 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:46.355175 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:46.347891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:50801)
0812 11:24:46.348039 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:46.348076 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:46.348148 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:46.348507 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:46.348510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:46.348803 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:46.348981 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.348992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.349997 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.350000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.351043 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.351055 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.351662 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.351666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.351802 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.352281 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:46.352297 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:46.354890 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 11:24:46.355358 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:46.348029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:50801 (local address 127.12.110.254:37905)
0812 11:24:46.348257 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:46.348261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:46.348275 (+ 14us) server_negotiation.cc:407] Connection header received
0812 11:24:46.348319 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:46.348322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:46.348382 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:46.348458 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:46.349126 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.349883 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.351162 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.351545 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.351636 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.355090 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:50801: BlockingRecv error: recv got EOF from 127.12.110.195:50801 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
W20260812 11:24:46.355540 28251 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:46.408201 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:46.399129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:49899)
0812 11:24:46.399638 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:46.399654 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:46.399770 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:46.400113 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:46.400116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:46.400588 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:46.400797 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.400806 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.401831 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.401835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.402894 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.402905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.403640 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.403644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.403762 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.404392 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:46.404415 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:46.407854 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":297,"threads_started":1}
W20260812 11:24:46.408340 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:46.399193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:49899 (local address 127.12.110.254:37905)
0812 11:24:46.399416 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:46.399426 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:46.399738 (+ 312us) server_negotiation.cc:407] Connection header received
0812 11:24:46.399897 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:46.399900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:46.399963 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:46.400037 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:46.400968 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.401691 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.403045 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.403477 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.403586 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.408103 (+ 4517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:49899: BlockingRecv error: recv got EOF from 127.12.110.193:49899 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:46 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:46.883936 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:46.874604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:33069)
0812 11:24:46.875224 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:46.875241 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:46.875439 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:46.875761 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:46.875764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:46.876269 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:46.876503 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.876513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.877540 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.877545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.878642 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:46.878654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.879402 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.879407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:46.879536 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.880151 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:46.880169 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:46.883603 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":282,"threads_started":1}
W20260812 11:24:46.884079 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:46.874662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:33069 (local address 127.12.110.254:37905)
0812 11:24:46.874898 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:46.874908 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:46.875380 (+ 472us) server_negotiation.cc:407] Connection header received
0812 11:24:46.875554 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:46.875558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:46.875623 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:46.875708 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:46.876649 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.877397 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.878780 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:46.879187 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:46.879280 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:46.883816 (+ 4536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:33069: BlockingRecv error: recv got EOF from 127.12.110.194:33069 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 12 11:24:47 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:47 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:47.365163 28265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:47.356284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:33397)
0812 11:24:47.356867 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:47.356901 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:47.357085 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:47.357459 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:47.357464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:47.358095 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:47.358292 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:47.358302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.359413 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.359419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:47.360406 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:47.360419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.361172 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.361179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:47.361367 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:47.361954 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:47.361971 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:47.364837 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":294,"threads_started":1}
W20260812 11:24:47.365304 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:47.356415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:33397 (local address 127.12.110.254:37905)
0812 11:24:47.356658 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:47.356664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:47.357025 (+ 361us) server_negotiation.cc:407] Connection header received
0812 11:24:47.357215 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:47.357227 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:47.357292 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:47.357422 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:47.358447 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.359249 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.360556 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.360975 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.361116 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:47.365049 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:33397: BlockingRecv error: recv got EOF from 127.12.110.195:33397 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 12 11:24:47 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:47 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:47.417806 28266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:47.409128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:54751)
0812 11:24:47.409628 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:47.409644 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:47.409760 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:47.410086 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:47.410089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:47.410508 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:47.410783 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:47.410793 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.411843 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.411848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:47.412763 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:47.412774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.413454 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.413471 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:47.413582 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:47.414166 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:47.414192 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:47.417443 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":267,"threads_started":1}
W20260812 11:24:47.417900 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:47.409264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:54751 (local address 127.12.110.254:37905)
0812 11:24:47.409511 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:47.409517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:47.409724 (+ 207us) server_negotiation.cc:407] Connection header received
0812 11:24:47.409856 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:47.409860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:47.409925 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:47.410006 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:47.410940 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.411703 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.412890 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.413300 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.413402 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:47.417645 (+ 4243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:54751: BlockingRecv error: recv got EOF from 127.12.110.193:54751 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Aug 12 11:24:47 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:47 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:47.894219 28267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:47.884919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:54303)
0812 11:24:47.885438 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:47.885454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:47.885576 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:47.886072 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:47.886077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:47.886680 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:47.886869 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:47.886882 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.888009 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.888013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:47.888930 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:47.888942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.889935 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.889939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:47.890055 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:47.890735 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:47.890753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:47.893968 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":298,"threads_started":1}
W20260812 11:24:47.894438 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:47.885110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:54303 (local address 127.12.110.254:37905)
0812 11:24:47.885332 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:47.885339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:47.885531 (+ 192us) server_negotiation.cc:407] Connection header received
0812 11:24:47.885695 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:47.885701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:47.885785 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:47.885870 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:47.887051 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.887830 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.889176 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:47.889746 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:47.889900 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:47.894200 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:54303: BlockingRecv error: recv got EOF from 127.12.110.194:54303 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 11:24:48 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:48 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:48.377647 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:48.366450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:58999)
0812 11:24:48.367353 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:48.367375 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:48.367563 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:48.368302 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:48.368307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:48.368942 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:48.369208 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:48.369219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.370484 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.370490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:48.372095 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:48.372108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.373000 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.373005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:48.373122 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:48.373746 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:48.373764 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:48.377400 (+ 3636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":594,"threads_started":1}
W20260812 11:24:48.378064 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:48.366769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:58999 (local address 127.12.110.254:37905)
0812 11:24:48.367172 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:48.367180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:48.367477 (+ 297us) server_negotiation.cc:407] Connection header received
0812 11:24:48.367767 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:48.367771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:48.368123 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:48.368255 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:48.369385 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.370273 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.372257 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.372744 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.372971 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:48.377592 (+ 4621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:58999: BlockingRecv error: recv got EOF from 127.12.110.195:58999 (error 108)
Metrics: {"server-negotiator.queue_time_us":207}
Aug 12 11:24:48 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:48 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:48.428301 28269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:48.419425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:56425)
0812 11:24:48.420076 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:48.420099 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:48.420250 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:48.420626 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:48.420633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:48.421036 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:48.421204 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:48.421213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.422276 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.422285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:48.423517 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:48.423531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.424284 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.424289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:48.424482 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:48.425128 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:48.425146 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:48.428005 (+ 2859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"spinlock_wait_cycles":90368,"thread_start_us":265,"threads_started":1}
W20260812 11:24:48.428400 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:48.419488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:56425 (local address 127.12.110.254:37905)
0812 11:24:48.419720 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:48.419729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:48.420173 (+ 444us) server_negotiation.cc:407] Connection header received
0812 11:24:48.420396 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:48.420401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:48.420481 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:48.420585 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:48.421369 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.422135 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.423686 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.424123 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.424258 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:48.428207 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:56425: BlockingRecv error: recv got EOF from 127.12.110.193:56425 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 11:24:48 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:48 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:48.904506 28270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:48.895204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:46733)
0812 11:24:48.896110 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:48.896132 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:48.896287 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:48.896690 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:48.896693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:48.897043 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:48.897217 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:48.897225 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.898424 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.898428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:48.899352 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:48.899363 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.900076 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.900081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:48.900213 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:48.900830 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:48.900848 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:48.904169 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":785,"thread_start_us":626,"threads_started":1}
W20260812 11:24:48.904956 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:48.895326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:46733 (local address 127.12.110.254:37905)
0812 11:24:48.895504 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:48.895510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:48.896209 (+ 699us) server_negotiation.cc:407] Connection header received
0812 11:24:48.896471 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:48.896476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:48.896564 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:48.896655 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:48.897374 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.898281 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.899502 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:48.899911 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:48.900010 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:48.904405 (+ 4395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:46733: BlockingRecv error: recv got EOF from 127.12.110.194:46733 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 12 11:24:49 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:49 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:49.387181 28271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:49.378671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:39175)
0812 11:24:49.379227 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:49.379243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:49.379356 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:49.379689 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:49.379693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:49.380059 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:49.380216 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:49.380224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.381626 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.381630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:49.382506 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:49.382517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.383318 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.383323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:49.383451 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:49.384043 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:49.384060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:49.386928 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":330,"threads_started":1}
W20260812 11:24:49.387339 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:49.378859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:39175 (local address 127.12.110.254:37905)
0812 11:24:49.379040 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:49.379044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:49.379317 (+ 273us) server_negotiation.cc:407] Connection header received
0812 11:24:49.379456 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:49.379459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:49.379536 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:49.379621 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:49.380351 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.381491 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.382720 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.383140 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.383262 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:49.387118 (+ 3856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:39175: BlockingRecv error: recv got EOF from 127.12.110.195:39175 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 12 11:24:49 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:49 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:49.437297 28273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:49.429376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:55269)
0812 11:24:49.429861 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:49.429878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:49.429997 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:49.430269 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:49.430272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:49.430728 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:49.430896 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:49.430905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.431944 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.431948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:49.432919 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:49.432930 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.433623 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.433627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:49.433743 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:49.434302 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:49.434319 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:49.437073 (+ 2754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":275,"threads_started":1}
W20260812 11:24:49.437539 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:49.429443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:55269 (local address 127.12.110.254:37905)
0812 11:24:49.429697 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:49.429703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:49.429960 (+ 257us) server_negotiation.cc:407] Connection header received
0812 11:24:49.430080 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:49.430084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:49.430143 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:49.430219 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:49.431055 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.431814 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.433059 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.433483 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.433576 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:49.437267 (+ 3691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:55269: BlockingRecv error: recv got EOF from 127.12.110.193:55269 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 12 11:24:49 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:49 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:49.917352 28274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:49.905694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:59693)
0812 11:24:49.906417 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:49.906437 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:49.906648 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:49.907023 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:49.907027 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:49.907465 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:49.907610 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:49.907618 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.908645 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.908653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:49.909616 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:49.909626 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.910326 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.910341 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:49.910518 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:49.914279 (+ 3761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:49.914298 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:49.917117 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":476,"threads_started":1}
W20260812 11:24:49.917526 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:49.905805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:59693 (local address 127.12.110.254:37905)
0812 11:24:49.906255 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:49.906263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:49.906588 (+ 325us) server_negotiation.cc:407] Connection header received
0812 11:24:49.906812 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:49.906816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:49.906885 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:49.906991 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:49.907756 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.908515 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.909760 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:49.910185 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:49.910312 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:49.917294 (+ 6982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:59693: BlockingRecv error: recv got EOF from 127.12.110.194:59693 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Aug 12 11:24:50 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:50 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:50.397539 28275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:50.388162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:39001)
0812 11:24:50.388659 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:50.388677 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:50.388822 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:50.389135 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:50.389138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:50.389535 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:50.389696 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:50.389706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.391386 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.391390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:50.392374 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:50.392386 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.393103 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.393108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:50.393236 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:50.393973 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:50.394027 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:50.397232 (+ 3205us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
W20260812 11:24:50.397802 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:50.388351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:39001 (local address 127.12.110.254:37905)
0812 11:24:50.388576 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:50.388584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:50.388758 (+ 174us) server_negotiation.cc:407] Connection header received
0812 11:24:50.388924 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:50.388927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:50.389001 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:50.389082 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:50.389857 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.391222 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.392532 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.392945 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.393051 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:50.397478 (+ 4427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:39001: BlockingRecv error: recv got EOF from 127.12.110.195:39001 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 12 11:24:50 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:50 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:50.448534 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:50.438382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:60049)
0812 11:24:50.438930 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:50.438951 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:50.439064 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:50.439566 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:50.439573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:50.440376 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:50.440585 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:50.440597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.441647 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.441651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:50.442613 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:50.442625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.443314 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.443318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:50.443436 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:50.444056 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:50.444074 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:50.447932 (+ 3858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":323,"threads_started":1}
W20260812 11:24:50.448658 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:50.438461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:60049 (local address 127.12.110.254:37905)
0812 11:24:50.438717 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:50.438727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:50.439051 (+ 324us) server_negotiation.cc:407] Connection header received
0812 11:24:50.439196 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:50.439200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:50.439265 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:50.439353 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:50.440740 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.441495 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.442751 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.443179 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.443273 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:50.448246 (+ 4973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:60049: BlockingRecv error: recv got EOF from 127.12.110.193:60049 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 12 11:24:50 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:50 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:50.928210 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:50.918457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:41465)
0812 11:24:50.919050 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:50.919066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:50.919223 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:50.919595 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:50.919602 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:50.920050 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:50.920228 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:50.920237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.921306 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.921314 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:50.922219 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:50.922231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.923560 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.923567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:50.923754 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:50.924531 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:50.924549 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:50.927806 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":310,"threads_started":1}
W20260812 11:24:50.928390 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:50.918658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:41465 (local address 127.12.110.254:37905)
0812 11:24:50.918847 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:50.918856 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:50.919175 (+ 319us) server_negotiation.cc:407] Connection header received
0812 11:24:50.919363 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:50.919368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:50.919441 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:50.919554 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:50.920385 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.921150 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.922657 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:50.923369 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:50.923548 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:50.928047 (+ 4499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:41465: BlockingRecv error: recv got EOF from 127.12.110.194:41465 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 11:24:51 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:51 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:51.409121 28278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:51.398824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:59921)
0812 11:24:51.399647 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:51.399668 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:51.399900 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:51.400353 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:51.400358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:51.400919 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:51.401106 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:51.401115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.402487 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.402494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:51.403516 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:51.403531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.404728 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.404734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:51.404922 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:51.405744 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:51.405772 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:51.408768 (+ 2996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":457,"threads_started":1}
W20260812 11:24:51.409253 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:51.399128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:59921 (local address 127.12.110.254:37905)
0812 11:24:51.399422 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:51.399433 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:51.399803 (+ 370us) server_negotiation.cc:407] Connection header received
0812 11:24:51.400064 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:51.400070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:51.400182 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:51.400311 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:51.401299 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.402272 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.403686 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.404562 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.404714 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:51.408978 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:59921: BlockingRecv error: recv got EOF from 127.12.110.195:59921 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Aug 12 11:24:51 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:51 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:51.459532 28281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:51.449877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:34531)
0812 11:24:51.450593 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:51.450612 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:51.450758 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:51.451188 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:51.451191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:51.451696 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:51.451891 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:51.451901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.453240 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.453246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:51.454460 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:51.454479 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.455457 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.455463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:51.455648 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:51.456278 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:51.456296 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:51.459303 (+ 3007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":406,"threads_started":1}
W20260812 11:24:51.459805 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:51.449920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:34531 (local address 127.12.110.254:37905)
0812 11:24:51.450281 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:51.450288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:51.450732 (+ 444us) server_negotiation.cc:407] Connection header received
0812 11:24:51.450927 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:51.450932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:51.451014 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:51.451149 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:51.452076 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.452906 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.454661 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.455279 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.455415 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:51.459515 (+ 4100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:34531: BlockingRecv error: recv got EOF from 127.12.110.193:34531 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 11:24:51 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:51 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:51.937551 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:51.929364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:33827)
0812 11:24:51.929867 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:51.929883 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:51.929992 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:51.930307 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:51.930310 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:51.930727 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:51.930881 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:51.930890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.931890 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.931894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:51.932770 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:51.932780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.933430 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.933434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:51.933548 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:51.934179 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:51.934196 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:51.937233 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":291,"threads_started":1}
W20260812 11:24:51.937733 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:51.929465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:33827 (local address 127.12.110.254:37905)
0812 11:24:51.929782 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:51.929789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:51.929971 (+ 182us) server_negotiation.cc:407] Connection header received
0812 11:24:51.930103 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:51.930107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:51.930175 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:51.930267 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:51.931032 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.931769 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.932898 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:51.933299 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:51.933389 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:51.937479 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:33827: BlockingRecv error: recv got EOF from 127.12.110.194:33827 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 12 11:24:52 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:52 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:52.419817 28283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:52.410260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:46613)
0812 11:24:52.410848 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:52.410864 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:52.411021 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:52.411390 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:52.411396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:52.411873 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:52.412038 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:52.412047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.413090 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.413096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:52.413991 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:52.414002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.414804 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.414810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:52.414988 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:52.415965 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:52.415984 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:52.419491 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":341,"threads_started":1}
W20260812 11:24:52.420015 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:52.410352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:46613 (local address 127.12.110.254:37905)
0812 11:24:52.410573 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:52.410579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:52.410969 (+ 390us) server_negotiation.cc:407] Connection header received
0812 11:24:52.411154 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:52.411159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:52.411246 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:52.411353 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:52.412194 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.412938 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.414141 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.414648 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.414780 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:52.419703 (+ 4923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:46613: BlockingRecv error: recv got EOF from 127.12.110.195:46613 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Aug 12 11:24:52 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:52 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:52.469048 28284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:52.460440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:59591)
0812 11:24:52.461054 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:52.461070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:52.461235 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:52.461607 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:52.461612 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:52.462038 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:52.462209 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:52.462217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.463351 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.463357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:52.464288 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:52.464300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.465024 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.465031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:52.465201 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:52.465815 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:52.465834 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:52.468697 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":273,"threads_started":1}
W20260812 11:24:52.469148 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:52.460538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:59591 (local address 127.12.110.254:37905)
0812 11:24:52.460755 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:52.460765 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:52.461167 (+ 402us) server_negotiation.cc:407] Connection header received
0812 11:24:52.461366 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:52.461373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:52.461447 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:52.461556 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:52.462359 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.463199 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.464444 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.464860 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.465007 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:52.468901 (+ 3894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:59591: BlockingRecv error: recv got EOF from 127.12.110.193:59591 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 12 11:24:52 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:52 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:52.947428 28285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:52.938615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:35491)
0812 11:24:52.939295 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:52.939315 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:52.939440 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:52.939747 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:52.939750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:52.940217 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:52.940402 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:52.940412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.941535 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.941540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:52.942479 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:52.942491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.943260 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.943265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:52.943396 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:52.944100 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:52.944118 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:52.947056 (+ 2938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":333,"threads_started":1}
W20260812 11:24:52.947522 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:52.938787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:35491 (local address 127.12.110.254:37905)
0812 11:24:52.939011 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:52.939016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:52.939399 (+ 383us) server_negotiation.cc:407] Connection header received
0812 11:24:52.939546 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:52.939549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:52.939612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:52.939694 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:52.940575 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.941370 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.942694 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:52.943116 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:52.943210 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:52.947243 (+ 4033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:35491: BlockingRecv error: recv got EOF from 127.12.110.194:35491 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 12 11:24:53 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:53 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:53.429315 28286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:53.420914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:52187)
0812 11:24:53.421566 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:53.421583 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:53.421696 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:53.421993 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:53.421996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:53.422432 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:53.422671 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:53.422682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.423623 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.423627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:53.424555 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:53.424566 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.425255 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.425259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:53.425370 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:53.425919 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:53.425936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:53.428972 (+ 3036us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":321,"threads_started":1}
W20260812 11:24:53.429487 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:53.420987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:52187 (local address 127.12.110.254:37905)
0812 11:24:53.421171 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:53.421176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:53.421687 (+ 511us) server_negotiation.cc:407] Connection header received
0812 11:24:53.421795 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:53.421798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:53.421872 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:53.421948 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:53.422806 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.423495 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.424686 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.425123 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.425215 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:53.429211 (+ 3996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:52187: BlockingRecv error: recv got EOF from 127.12.110.195:52187 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 12 11:24:53 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:53 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:53.479151 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:53.470054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:57733)
0812 11:24:53.470714 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:53.470730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:53.470948 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:53.471313 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:53.471317 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:53.471778 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:53.471980 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:53.471990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.472991 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.472997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:53.474093 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:53.474105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.474986 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.474993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:53.475164 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:53.475881 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:53.475904 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:53.478905 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":287,"threads_started":1}
W20260812 11:24:53.479478 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:53.470298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:57733 (local address 127.12.110.254:37905)
0812 11:24:53.470498 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:53.470504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:53.470834 (+ 330us) server_negotiation.cc:407] Connection header received
0812 11:24:53.471089 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:53.471094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:53.471174 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:53.471284 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:53.472119 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.472841 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.474229 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.474749 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.474910 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:53.479100 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:57733: BlockingRecv error: recv got EOF from 127.12.110.193:57733 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 12 11:24:53 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:53 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:53.957454 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:53.948917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:44967)
0812 11:24:53.949577 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:53.949597 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:53.949739 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:53.950121 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:53.950125 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:53.950550 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:53.950702 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:53.950710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.951822 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.951825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:53.952717 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:53.952727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.953400 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.953404 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:53.953524 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:53.954222 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:53.954240 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:53.957207 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":355,"threads_started":1}
W20260812 11:24:53.957667 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:53.948965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:44967 (local address 127.12.110.254:37905)
0812 11:24:53.949210 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:53.949220 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:53.949715 (+ 495us) server_negotiation.cc:407] Connection header received
0812 11:24:53.949884 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:53.949892 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:53.949980 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:53.950082 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:53.950821 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.951700 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.952841 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:53.953241 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:53.953341 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:53.957438 (+ 4097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:44967: BlockingRecv error: recv got EOF from 127.12.110.194:44967 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 12 11:24:54 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:54 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:54.439116 28290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:54.430281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:34597)
0812 11:24:54.430866 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:54.430883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:54.430993 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:54.431304 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:54.431306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:54.431703 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:54.431867 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:54.431876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.432980 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.432986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:54.434179 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:54.434194 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.434970 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.434976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:54.435138 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:54.435734 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:54.435751 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:54.438869 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":324,"threads_started":1}
W20260812 11:24:54.439324 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:54.430412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:34597 (local address 127.12.110.254:37905)
0812 11:24:54.430668 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:54.430673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:54.430955 (+ 282us) server_negotiation.cc:407] Connection header received
0812 11:24:54.431100 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:54.431104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:54.431175 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:54.431259 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:54.431999 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.432816 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.434326 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.434770 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.434927 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:54.439057 (+ 4130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:34597: BlockingRecv error: recv got EOF from 127.12.110.195:34597 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 12 11:24:54 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:54 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:54.489591 28291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:54.480046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:50235)
0812 11:24:54.480656 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:54.480677 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:54.480825 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:54.481178 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:54.481183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:54.481719 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:54.481895 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:54.481905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.482986 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.482991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:54.483948 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:54.483960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.484712 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.484719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:54.484893 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:54.485528 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:54.485555 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:54.489234 (+ 3679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":336,"threads_started":1}
W20260812 11:24:54.489670 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:54.480084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:50235 (local address 127.12.110.254:37905)
0812 11:24:54.480291 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:54.480299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:54.480794 (+ 495us) server_negotiation.cc:407] Connection header received
0812 11:24:54.480957 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:54.480962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:54.481045 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:54.481152 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:54.482045 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.482846 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.484092 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.484536 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.484680 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:54.489441 (+ 4761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:50235: BlockingRecv error: recv got EOF from 127.12.110.193:50235 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 12 11:24:54 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:54 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:54.970507 28293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:54.958524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:45125)
0812 11:24:54.959169 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:54.959186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:54.959364 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:54.959736 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:54.959741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:54.960199 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:54.960386 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:54.960396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.961504 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.961511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:54.962487 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:54.962498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.963326 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.963333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:54.963522 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:54.966858 (+ 3336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:54.966878 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:54.970069 (+ 3191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":348,"threads_started":1}
W20260812 11:24:54.970731 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:54.958610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:45125 (local address 127.12.110.254:37905)
0812 11:24:54.958793 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:54.958799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:54.959283 (+ 484us) server_negotiation.cc:407] Connection header received
0812 11:24:54.959509 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:54.959519 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:54.959587 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:54.959705 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:54.960545 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.961348 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.962690 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:54.963129 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:54.963278 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:54.970355 (+ 7077us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:45125: BlockingRecv error: recv got EOF from 127.12.110.194:45125 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 12 11:24:55 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:55 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:55.449082 28294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:55.440162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:39771)
0812 11:24:55.440707 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:55.440723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:55.440897 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:55.441256 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:55.441261 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:55.441729 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:55.441914 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:55.441923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.443000 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.443007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:55.443954 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:55.443965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.444726 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.444732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:55.444915 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:55.445498 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:55.445515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:55.448730 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":296,"threads_started":1}
W20260812 11:24:55.449190 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:55.440244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:39771 (local address 127.12.110.254:37905)
0812 11:24:55.440542 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:55.440552 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:55.440821 (+ 269us) server_negotiation.cc:407] Connection header received
0812 11:24:55.441032 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:55.441042 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:55.441118 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:55.441225 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:55.442065 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.442867 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.444103 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.444541 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.444695 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:55.448939 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:39771: BlockingRecv error: recv got EOF from 127.12.110.195:39771 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Aug 12 11:24:55 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:55 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:55.499636 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:55.490634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:55179)
0812 11:24:55.491221 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:55.491236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:55.491395 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:55.491747 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:55.491751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:55.492172 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:55.492315 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:55.492323 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.493391 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.493397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:55.494347 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:55.494359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.495444 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.495475 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:55.495703 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:55.496464 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:55.496481 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:55.499379 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":380,"threads_started":1}
W20260812 11:24:55.499846 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:55.490717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:55179 (local address 127.12.110.254:37905)
0812 11:24:55.490968 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:55.490976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:55.491326 (+ 350us) server_negotiation.cc:407] Connection header received
0812 11:24:55.491523 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:55.491528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:55.491596 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:55.491714 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:55.492474 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.493247 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.494490 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.495263 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.495398 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:55.499600 (+ 4202us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:55179: BlockingRecv error: recv got EOF from 127.12.110.193:55179 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 12 11:24:55 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:55 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:55.980059 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:55.971643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:60117)
0812 11:24:55.972224 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:55.972242 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:55.972429 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:55.972735 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:55.972739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:55.973169 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:55.973356 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:55.973367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.974497 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.974502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:55.975640 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:55.975653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.976365 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.976369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:55.976501 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:55.977079 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:55.977096 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:55.979836 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":343,"threads_started":1}
W20260812 11:24:55.980281 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:55.971770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:60117 (local address 127.12.110.254:37905)
0812 11:24:55.971951 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:55.971956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:55.972350 (+ 394us) server_negotiation.cc:407] Connection header received
0812 11:24:55.972538 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:55.972541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:55.972604 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:55.972690 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:55.973520 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.974339 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.975775 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:55.976204 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:55.976300 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:55.980036 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:60117: BlockingRecv error: recv got EOF from 127.12.110.194:60117 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 12 11:24:56 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:56 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:56.458230 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:56.450268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:43797)
0812 11:24:56.450863 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:56.450880 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:56.450985 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:56.451282 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:56.451285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:56.451657 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:56.451814 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:56.451822 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.452770 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.452773 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:56.453693 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:56.453704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.454362 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.454366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:56.454487 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:56.455219 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:56.455237 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:56.458012 (+ 2775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":373,"threads_started":1}
W20260812 11:24:56.458544 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:56.450392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:43797 (local address 127.12.110.254:37905)
0812 11:24:56.450628 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:56.450639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:56.450965 (+ 326us) server_negotiation.cc:407] Connection header received
0812 11:24:56.451078 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:56.451081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:56.451155 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:56.451237 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:56.451954 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.452655 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.453827 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.454222 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.454311 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:56.458219 (+ 3908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:43797: BlockingRecv error: recv got EOF from 127.12.110.195:43797 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 12 11:24:56 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:56 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:56.509150 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:56.500582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:44883)
0812 11:24:56.501121 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:56.501138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:56.501249 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:56.501672 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:56.501676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:56.502144 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:56.502334 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:56.502342 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.503389 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.503393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:56.504302 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:56.504314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.505018 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.505023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:56.505154 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:56.505770 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:56.505787 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:56.508789 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":308,"threads_started":1}
W20260812 11:24:56.509272 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:56.500745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:44883 (local address 127.12.110.254:37905)
0812 11:24:56.500931 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:56.500936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:56.501220 (+ 284us) server_negotiation.cc:407] Connection header received
0812 11:24:56.501414 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:56.501418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:56.501489 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:56.501578 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:56.502506 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.503243 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.504453 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.504886 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.504986 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:56.509001 (+ 4015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:44883: BlockingRecv error: recv got EOF from 127.12.110.193:44883 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 12 11:24:56 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:56 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:56.989763 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:56.981010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:43553)
0812 11:24:56.981549 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:56.981565 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:56.981733 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:56.982096 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:56.982101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:56.982572 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:56.982752 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:56.982760 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.983801 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.983807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:56.984701 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:56.984712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.985497 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.985503 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:56.985681 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:56.986389 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:56.986409 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:56.989420 (+ 3011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":290,"threads_started":1}
W20260812 11:24:56.989929 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:56.981097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:43553 (local address 127.12.110.254:37905)
0812 11:24:56.981371 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:56.981378 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:56.981670 (+ 292us) server_negotiation.cc:407] Connection header received
0812 11:24:56.981874 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:56.981879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:56.981952 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:56.982066 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:56.982903 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.983658 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.984848 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:56.985334 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:56.985466 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:56.989613 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:43553: BlockingRecv error: recv got EOF from 127.12.110.194:43553 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 12 11:24:57 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:57 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:57.470198 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:57.459253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:51969)
0812 11:24:57.460043 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:57.460073 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:57.460428 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:57.460815 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:57.460820 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:57.461297 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:57.461483 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:57.461492 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.462564 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.462574 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:57.463588 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:57.463601 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.464819 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.464830 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:57.465054 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:57.466007 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:57.466027 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:57.469760 (+ 3733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":325,"threads_started":1}
W20260812 11:24:57.470315 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:57.459437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:51969 (local address 127.12.110.254:37905)
0812 11:24:57.459697 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:57.459703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:57.460309 (+ 606us) server_negotiation.cc:407] Connection header received
0812 11:24:57.460588 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:57.460594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:57.460667 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:57.460781 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:57.461667 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.462398 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.463758 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.464478 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.464744 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:57.470023 (+ 5279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:51969: BlockingRecv error: recv got EOF from 127.12.110.195:51969 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Aug 12 11:24:57 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:57 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:57.519728 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:57.510771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:33801)
0812 11:24:57.511393 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:57.511411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:57.511586 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:57.511953 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:57.511960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:57.512431 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:57.512580 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:57.512588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.513641 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.513647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:57.514585 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:57.514596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.515365 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.515371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:57.515558 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:57.516245 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:57.516263 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:57.519510 (+ 3247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":359,"threads_started":1}
W20260812 11:24:57.520161 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:57.510874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:33801 (local address 127.12.110.254:37905)
0812 11:24:57.511205 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:57.511213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:57.511510 (+ 297us) server_negotiation.cc:407] Connection header received
0812 11:24:57.511720 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:57.511725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:57.511796 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:57.511908 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:57.512730 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.513482 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.514728 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.515204 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.515347 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:57.519747 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:33801: BlockingRecv error: recv got EOF from 127.12.110.193:33801 (error 108)
Metrics: {"server-negotiator.queue_time_us":158}
Aug 12 11:24:57 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:57 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:57.999639 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:57.990901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:57517)
0812 11:24:57.991440 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:57.991457 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:57.991583 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:57.991902 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:57.991905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:57.992341 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:57.992513 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:57.992522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.993613 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.993617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:57.994526 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:57.994607 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.995371 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.995377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:57.995584 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:57.996302 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:57.996329 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:57.999382 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":316,"threads_started":1}
W20260812 11:24:57.999716 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:57.991026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:57517 (local address 127.12.110.254:37905)
0812 11:24:57.991293 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:57.991300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:57.991542 (+ 242us) server_negotiation.cc:407] Connection header received
0812 11:24:57.991683 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:57.991686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:57.991749 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:57.991829 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:57.992685 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.993417 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.994751 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:57.995198 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:57.995345 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:57.999532 (+ 4187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:57517: BlockingRecv error: recv got EOF from 127.12.110.194:57517 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 12 11:24:58 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:58 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:58.481850 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:58.471463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:41633)
0812 11:24:58.472124 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:58.472141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:58.472257 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:58.472698 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:58.472702 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:58.473184 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:58.473342 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:58.473350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.475105 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.475109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:58.476382 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:58.476394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.477570 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.477576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:58.477718 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:58.478433 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:58.478454 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:58.481609 (+ 3155us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":422,"threads_started":1}
W20260812 11:24:58.483014 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:58.471462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:41633 (local address 127.12.110.254:37905)
0812 11:24:58.471744 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:58.471753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:58.472226 (+ 473us) server_negotiation.cc:407] Connection header received
0812 11:24:58.472394 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:58.472401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:58.472502 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:58.472643 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:58.473529 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.474941 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.476545 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.477406 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.477558 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:58.482483 (+ 4925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:41633: BlockingRecv error: recv got EOF from 127.12.110.195:41633 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 12 11:24:58 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:58 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:58.528760 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:58.520677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:34169)
0812 11:24:58.521210 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:58.521226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:58.521333 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:58.521611 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:58.521614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:58.522082 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:58.522264 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:58.522273 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.523301 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.523305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:58.524220 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:58.524231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.524916 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.524920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:58.525036 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:58.525658 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:58.525675 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:58.528530 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":293,"threads_started":1}
W20260812 11:24:58.528973 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:58.520761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:34169 (local address 127.12.110.254:37905)
0812 11:24:58.521033 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:58.521040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:58.521304 (+ 264us) server_negotiation.cc:407] Connection header received
0812 11:24:58.521422 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:58.521426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:58.521491 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:58.521571 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:58.522386 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.523175 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.524355 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:58.524766 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:58.524858 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:58.528726 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:34169: BlockingRecv error: recv got EOF from 127.12.110.193:34169 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 11:24:59 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:59 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:59.011796 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:59.000745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:33535)
0812 11:24:59.001285 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:59.001301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:59.001424 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:59.001714 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:59.001717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:59.002115 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:59.002284 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:59.002292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.003862 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.003867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:59.005022 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:59.005036 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.006229 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.006237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:59.006387 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:59.007107 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:59.007126 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:59.011551 (+ 4425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":290,"threads_started":1}
W20260812 11:24:59.012337 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:59.000941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:33535 (local address 127.12.110.254:37905)
0812 11:24:59.001176 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:59.001182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:59.001435 (+ 253us) server_negotiation.cc:407] Connection header received
0812 11:24:59.001515 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:59.001518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:59.001590 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:59.001673 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:59.002443 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.003590 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.005298 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.005971 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.006132 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:59.011996 (+ 5864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:33535: BlockingRecv error: recv got EOF from 127.12.110.194:33535 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 12 11:24:59 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:59 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:59.492177 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:59.483034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:59701)
0812 11:24:59.483614 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:59.483631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:59.483808 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:59.484164 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:59.484169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:59.484619 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:59.484776 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:59.484785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.485872 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.485878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:59.486915 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:59.486927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.487733 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.487740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:59.487877 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:59.488678 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:59.488705 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:59.491818 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":326,"threads_started":1}
W20260812 11:24:59.492254 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:59.483153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:59701 (local address 127.12.110.254:37905)
0812 11:24:59.483421 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:59.483431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:59.483724 (+ 293us) server_negotiation.cc:407] Connection header received
0812 11:24:59.483946 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:59.483958 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:59.484022 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:59.484135 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:59.484960 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.485729 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.487078 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.487493 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.487635 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:59.492019 (+ 4384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:59701: BlockingRecv error: recv got EOF from 127.12.110.195:59701 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 12 11:24:59 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:24:59 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:24:59.539022 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:24:59.529854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:39961)
0812 11:24:59.530377 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0812 11:24:59.530394 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:24:59.530513 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:24:59.530906 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:24:59.530909 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:24:59.531298 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:24:59.531491 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:59.531500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.532994 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.532998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:59.533936 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:24:59.533947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.534762 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.534767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:24:59.534920 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:59.535635 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:24:59.535653 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:24:59.538769 (+ 3116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":292,"threads_started":1}
W20260812 11:24:59.539184 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:24:59.529924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:39961 (local address 127.12.110.254:37905)
0812 11:24:59.530284 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0812 11:24:59.530291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:24:59.530510 (+ 219us) server_negotiation.cc:407] Connection header received
0812 11:24:59.530661 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:24:59.530665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:24:59.530729 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:24:59.530817 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:24:59.531677 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.532736 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.534106 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:24:59.534601 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:24:59.534701 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:24:59.538956 (+ 4255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:39961: BlockingRecv error: recv got EOF from 127.12.110.193:39961 (error 108)
Metrics: {"server-negotiator.queue_time_us":223}
Aug 12 11:25:00 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:25:00 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:25:00.023693 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:00.012893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:60781)
0812 11:25:00.013381 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:00.013398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:00.013511 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:00.013868 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:00.013872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:00.014265 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:00.014463 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:00.014472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.015548 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.015553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:00.016465 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:00.016476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.017183 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.017188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:00.017328 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:00.020521 (+ 3193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:25:00.020540 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:00.023446 (+ 2906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":260,"threads_started":1}
W20260812 11:25:00.023910 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:25:00.013048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:60781 (local address 127.12.110.254:37905)
0812 11:25:00.013280 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:00.013286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:00.013481 (+ 195us) server_negotiation.cc:407] Connection header received
0812 11:25:00.013607 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:00.013611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:00.013677 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:00.013752 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:00.014663 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.015410 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.016606 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.017031 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.017124 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:00.023657 (+ 6533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:60781: BlockingRecv error: recv got EOF from 127.12.110.194:60781 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 12 11:25:00 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:25:00 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.195@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:25:00.504503 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:00.493413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.195:34309)
0812 11:25:00.494299 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:00.494330 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:00.494786 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:00.495362 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:00.495367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:00.495838 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:00.496054 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:00.496064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.497500 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.497506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:00.498629 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:00.498652 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.499725 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.499732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:00.500105 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:00.500806 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:25:00.500825 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:00.504134 (+ 3309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":470,"threads_started":1}
W20260812 11:25:00.504659 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:25:00.493673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:34309 (local address 127.12.110.254:37905)
0812 11:25:00.494041 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:00.494050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:00.494641 (+ 591us) server_negotiation.cc:407] Connection header received
0812 11:25:00.494968 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:00.494974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:00.495064 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:00.495323 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:00.496233 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.497266 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.498811 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.499518 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.499703 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:00.504393 (+ 4690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.195:34309: BlockingRecv error: recv got EOF from 127.12.110.195:34309 (error 108)
Metrics: {"server-negotiator.queue_time_us":168}
Aug 12 11:25:00 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:25:00 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.193@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:25:00.549503 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:00.540409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.193:50383)
0812 11:25:00.540985 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:00.541002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:00.541110 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:00.541414 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:00.541417 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:00.541808 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:00.541986 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:00.541994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.543251 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.543256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:00.544217 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:00.544229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.545027 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.545032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:00.545163 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:00.545797 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:25:00.545815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:00.549159 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":328,"threads_started":1}
W20260812 11:25:00.549629 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:25:00.540464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:50383 (local address 127.12.110.254:37905)
0812 11:25:00.540661 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:00.540665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:00.541092 (+ 427us) server_negotiation.cc:407] Connection header received
0812 11:25:00.541209 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:00.541212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:00.541278 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:00.541374 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:00.542122 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.542915 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.544390 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:00.544852 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:00.544993 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:00.549405 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.193:50383: BlockingRecv error: recv got EOF from 127.12.110.193:50383 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 12 11:25:01 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
Aug 12 11:25:01 dist-test-slave-h6n0 krb5kdc[26441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.12.110.194@KRBTEST.COM for oryx/127.12.110.254@KRBTEST.COM, Server not found in Kerberos database
I20260812 11:25:01.033861 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:01.024913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37905 (local address 127.12.110.194:37487)
0812 11:25:01.025555 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:01.025571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:01.025687 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:01.026015 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:01.026019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:01.026445 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:01.026703 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:01.026713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.027907 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.027911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:01.028844 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:01.028854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.029609 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.029614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:01.029764 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:01.030622 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:25:01.030640 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:01.033558 (+ 2918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.12.110.254:37905: Server oryx/127.12.110.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":398,"threads_started":1}
W20260812 11:25:01.034397 27766 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:25:01.025240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:37487 (local address 127.12.110.254:37905)
0812 11:25:01.025497 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:01.025506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:01.025659 (+ 153us) server_negotiation.cc:407] Connection header received
0812 11:25:01.025788 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:01.025792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:01.025856 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:01.025944 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:01.026881 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.027712 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.029002 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.029450 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.029569 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:01.033973 (+ 4404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.12.110.194:37487: BlockingRecv error: recv got EOF from 127.12.110.194:37487 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 11:25:01.330883 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 27486
I20260812 11:25:01.342968 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:01.332037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42264 (local address 127.12.110.193:39827)
0812 11:25:01.332609 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:01.332616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:01.332643 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:25:01.332704 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:01.332709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:01.332923 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:01.333071 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:01.334510 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.335529 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.336732 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.337258 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.337371 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:01.339843 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:01.339851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:01.339864 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:01.339898 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:01.342431 (+ 2533us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.110.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 11:25:01.342741 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42264: Request ticket server kudu/127.12.110.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":275,"threads_started":1}
W20260812 11:25:01.343096 28335 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42264: Request ticket server kudu/127.12.110.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 11:25:01.343118 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 27486: Not authorized: Client connection negotiation failed: client connection to 127.12.110.193:39827: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.110.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 11:25:01.343317 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 27486
I20260812 11:25:01.368415 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 27771
I20260812 11:25:01.381309 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:01.369645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58260 (local address 127.12.110.194:33639)
0812 11:25:01.370338 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:01.370344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:01.370375 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:25:01.370475 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:01.370484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:01.370767 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:01.370913 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:01.372345 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.373408 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.374944 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.375663 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.375763 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:01.378132 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:01.378140 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:01.378153 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:01.378181 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:01.380737 (+ 2556us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.110.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 11:25:01.380972 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58260: Request ticket server kudu/127.12.110.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":369,"threads_started":1}
W20260812 11:25:01.381460 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 27771: Not authorized: Client connection negotiation failed: client connection to 127.12.110.194:33639: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.110.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 11:25:01.381537 28336 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58260: Request ticket server kudu/127.12.110.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 11:25:01.381628 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 27771
I20260812 11:25:01.408043 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 28006
I20260812 11:25:01.422597 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:01.409289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.12.110.195:41925)
0812 11:25:01.409963 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:01.409970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:01.409997 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:25:01.410061 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:01.410068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:01.410281 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:01.410435 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:01.411938 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.413081 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.414481 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:01.415139 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:01.415265 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:01.418708 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:01.418717 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:01.418732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:01.418768 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:01.421996 (+ 3228us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.110.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0812 11:25:01.422281 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46190: Request ticket server kudu/127.12.110.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":381,"threads_started":1}
W20260812 11:25:01.422719 28338 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46190: Request ticket server kudu/127.12.110.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260812 11:25:01.423041 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 28006: Not authorized: Client connection negotiation failed: client connection to 127.12.110.195:41925: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.12.110.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260812 11:25:01.423236 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 28006
I20260812 11:25:01.447149 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 27348
I20260812 11:25:01.448455 27474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:01.572178 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 27348
2026-08-12T11:25:01Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21979 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-12T11:25:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:25:01Z Disabled control of system clock
I20260812 11:25:01.676374 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40667
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:40667
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260812 11:25:02.030277 28346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:02.030665 28346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:02.030783 28346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:02.037999 28346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:25:02.038105 28346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:02.038143 28346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:02.038200 28346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:25:02.038245 28346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:25:02.048084 28346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:40667
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:02.049324 28346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:02.051093 28346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:02.063935 28365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:02.063971 28358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:02.063995 28360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:02.064656 28346 server_base.cc:1034] running on GCE node
I20260812 11:25:02.065443 28346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:02.066879 28346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:02.068120 28346 hybrid_clock.cc:648] HybridClock initialized: now 1786533902068084 us; error 53 us; skew 500 ppm
I20260812 11:25:02.068478 28346 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:02.070756 28346 webserver.cc:460] Webserver started at http://127.12.110.254:35365/ using document root <none> and password file <none>
I20260812 11:25:02.071432 28346 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:02.071581 28346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:02.071915 28346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:02.074827 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "5b609cb1cada4fc4b218a19c7421cb3c"
format_stamp: "Formatted at 2026-08-12 11:25:02 on dist-test-slave-h6n0"
server_key: "5070e63342e68f4a4f9e6b0681729068"
server_key_iv: "476ae3753a99d29804c161212f8e2621"
server_key_version: "encryptionkey@0"
I20260812 11:25:02.075646 28346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "5b609cb1cada4fc4b218a19c7421cb3c"
format_stamp: "Formatted at 2026-08-12 11:25:02 on dist-test-slave-h6n0"
server_key: "5070e63342e68f4a4f9e6b0681729068"
server_key_iv: "476ae3753a99d29804c161212f8e2621"
server_key_version: "encryptionkey@0"
I20260812 11:25:02.080641 28346 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:25:02.084213 28382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:02.085454 28346 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:25:02.085655 28346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "5b609cb1cada4fc4b218a19c7421cb3c"
format_stamp: "Formatted at 2026-08-12 11:25:02 on dist-test-slave-h6n0"
server_key: "5070e63342e68f4a4f9e6b0681729068"
server_key_iv: "476ae3753a99d29804c161212f8e2621"
server_key_version: "encryptionkey@0"
I20260812 11:25:02.085872 28346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:02.107388 28346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:02.108180 28346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:02.108467 28346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:02.127442 28346 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:40667
I20260812 11:25:02.127485 28470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:40667 every 8 connection(s)
I20260812 11:25:02.128836 28346 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:25:02.132584 28471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:25:02.137300 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 28346
I20260812 11:25:02.137529 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:25:02.138399 12731 external_mini_cluster.cc:1428] Setting key 7a5acc1968cca56065b4412cab58ba42
I20260812 11:25:02.145254 28471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c: Bootstrap starting.
I20260812 11:25:02.150091 28471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c: Neither blocks nor log segments found. Creating new log.
I20260812 11:25:02.151456 28471 log.cc:824] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:25:02.155246 28471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c: No bootstrap required, opened a new log
I20260812 11:25:02.158195 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:02.140872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34062 (local address 127.12.110.254:40667)
0812 11:25:02.142165 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:02.142181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:02.142843 (+ 662us) server_negotiation.cc:407] Connection header received
0812 11:25:02.146087 (+ 3244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:02.146120 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:02.146824 (+ 704us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:02.147312 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:02.148675 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:02.150747 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:02.152617 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:02.153568 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:02.153709 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:02.155377 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:02.155392 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:02.155413 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:02.155965 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:02.156044 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:02.156070 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:02.156157 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:02.156454 (+ 297us) server_negotiation.cc:299] Negotiation successful
0812 11:25:02.157127 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":432,"threads_started":1}
I20260812 11:25:02.163247 28471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40667 } }
I20260812 11:25:02.163585 28471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:25:02.163698 28471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b609cb1cada4fc4b218a19c7421cb3c, State: Initialized, Role: FOLLOWER
I20260812 11:25:02.164278 28471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40667 } }
I20260812 11:25:02.164494 28471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:25:02.164647 28471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:25:02.164830 28471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:25:02.167333 28471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40667 } }
I20260812 11:25:02.167845 28471 leader_election.cc:302] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b609cb1cada4fc4b218a19c7421cb3c; no voters:
I20260812 11:25:02.168593 28471 leader_election.cc:288] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:25:02.168993 28480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:25:02.170650 28480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [term 1 LEADER]: Becoming Leader. State: Replica: 5b609cb1cada4fc4b218a19c7421cb3c, State: Running, Role: LEADER
I20260812 11:25:02.171254 28471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:25:02.171262 28480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40667 } }
I20260812 11:25:02.175578 28482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b609cb1cada4fc4b218a19c7421cb3c. Latest consensus state: current_term: 1 leader_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40667 } } }
I20260812 11:25:02.175577 28481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b609cb1cada4fc4b218a19c7421cb3c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40667 } } }
I20260812 11:25:02.176124 28482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [sys.catalog]: This master's current role is: LEADER
I20260812 11:25:02.176126 28481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c [sys.catalog]: This master's current role is: LEADER
W20260812 11:25:02.190251 28509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:25:02.190424 28509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:25:02.190636 28499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:25:02.192840 28499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:25:02.202023 28499 catalog_manager.cc:1347] Generated new cluster ID: e031d8ae525b40c895921a2cc91c9136
I20260812 11:25:02.202165 28499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:25:02.209708 28499 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:25:02.211041 28499 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:25:02.223712 28499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b609cb1cada4fc4b218a19c7421cb3c: Generated new TSK 0
I20260812 11:25:02.224537 28499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:25:02.229945 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40667
--builtin_ntp_servers=127.12.110.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260812 11:25:02.603941 28517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:02.604207 28517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:02.604358 28517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:02.611824 28517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:02.611927 28517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:02.612105 28517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:25:02.621435 28517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:02.622776 28517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:02.624527 28517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:02.638362 28523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:02.638417 28529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:02.638445 28527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:02.639869 28517 server_base.cc:1034] running on GCE node
I20260812 11:25:02.640565 28517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:02.641623 28517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:02.642879 28517 hybrid_clock.cc:648] HybridClock initialized: now 1786533902642802 us; error 57 us; skew 500 ppm
I20260812 11:25:02.643230 28517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:02.645776 28517 webserver.cc:460] Webserver started at http://127.12.110.193:36025/ using document root <none> and password file <none>
I20260812 11:25:02.646705 28517 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:02.646859 28517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:02.647235 28517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:02.650198 28517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "03a68c7b569b4db98188b11f20881aa5"
format_stamp: "Formatted at 2026-08-12 11:25:02 on dist-test-slave-h6n0"
server_key: "451f8b4c3a6cb188fd0fb82e6ea6cd8a"
server_key_iv: "1c99873e5f0e7b887dcc918e81581324"
server_key_version: "encryptionkey@0"
I20260812 11:25:02.651049 28517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "03a68c7b569b4db98188b11f20881aa5"
format_stamp: "Formatted at 2026-08-12 11:25:02 on dist-test-slave-h6n0"
server_key: "451f8b4c3a6cb188fd0fb82e6ea6cd8a"
server_key_iv: "1c99873e5f0e7b887dcc918e81581324"
server_key_version: "encryptionkey@0"
I20260812 11:25:02.656958 28517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:25:02.660641 28540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:02.661990 28517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 11:25:02.662184 28517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "03a68c7b569b4db98188b11f20881aa5"
format_stamp: "Formatted at 2026-08-12 11:25:02 on dist-test-slave-h6n0"
server_key: "451f8b4c3a6cb188fd0fb82e6ea6cd8a"
server_key_iv: "1c99873e5f0e7b887dcc918e81581324"
server_key_version: "encryptionkey@0"
I20260812 11:25:02.662428 28517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:02.677677 28517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:02.678946 28517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:02.679382 28517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:02.680423 28517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:02.681957 28517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:02.682050 28517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:02.682225 28517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:02.682343 28517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:02.717724 28517 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:34663
I20260812 11:25:02.717803 28750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:34663 every 8 connection(s)
I20260812 11:25:02.719169 28517 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:25:02.727552 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 28517
I20260812 11:25:02.727749 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:25:02.728698 12731 external_mini_cluster.cc:1428] Setting key 6f35a16610469ba2d7259204448ce7a0
I20260812 11:25:02.731595 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40667
--builtin_ntp_servers=127.12.110.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 11:25:02.732762 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:02.721839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:60769 (local address 127.12.110.254:40667)
0812 11:25:02.722329 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:02.722334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:02.723350 (+ 1016us) server_negotiation.cc:407] Connection header received
0812 11:25:02.725164 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:02.725169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:02.725260 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:02.725396 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:02.726437 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:02.727217 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:02.729699 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:02.730102 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:02.730202 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:02.731890 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:02.731895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:02.731898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:02.731914 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:02.731951 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:02.731964 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:02.732071 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:02.732314 (+ 243us) server_negotiation.cc:299] Negotiation successful
0812 11:25:02.732410 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":271,"threads_started":1}
I20260812 11:25:02.733721 28758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:02.722222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40667 (local address 127.12.110.193:60769)
0812 11:25:02.723199 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:02.723235 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:02.724899 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:02.725631 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:02.725639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:02.725717 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:02.726257 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:02.726273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:02.727396 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:02.727400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:02.729338 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:02.729355 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:02.730370 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:02.730379 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:02.730711 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:02.731478 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:25:02.731480 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:02.731517 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:25:02.731587 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:02.732073 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:02.732087 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:02.732097 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:25:02.732301 (+ 204us) client_negotiation.cc:240] Negotiation successful
0812 11:25:02.732669 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":343,"threads_started":1}
I20260812 11:25:02.735206 28751 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40667
I20260812 11:25:02.735515 28751 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:02.736428 28751 heartbeater.cc:511] Master 127.12.110.254:40667 requested a full tablet report, sending...
I20260812 11:25:02.738689 28412 ts_manager.cc:194] Registered new tserver with Master: 03a68c7b569b4db98188b11f20881aa5 (127.12.110.193:34663)
I20260812 11:25:02.740795 28412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.193:60769
W20260812 11:25:03.089124 28762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:03.089443 28762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:03.089620 28762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:03.096843 28762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:03.096975 28762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:03.097138 28762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:25:03.106067 28762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:03.107374 28762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:03.109134 28762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:03.122404 28770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:03.122381 28771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:03.122928 28773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:03.123366 28762 server_base.cc:1034] running on GCE node
I20260812 11:25:03.123924 28762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:03.124790 28762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:03.126004 28762 hybrid_clock.cc:648] HybridClock initialized: now 1786533903125941 us; error 54 us; skew 500 ppm
I20260812 11:25:03.126292 28762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:03.129832 28762 webserver.cc:460] Webserver started at http://127.12.110.194:36647/ using document root <none> and password file <none>
I20260812 11:25:03.130589 28762 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:03.130713 28762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:03.131052 28762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:03.133940 28762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "266898dd7ac94e7dbf45c3df03e35b30"
format_stamp: "Formatted at 2026-08-12 11:25:03 on dist-test-slave-h6n0"
server_key: "abb3b5c392621372840ed9032a62e782"
server_key_iv: "596f9a67154961f5a3229f8f554b69cb"
server_key_version: "encryptionkey@0"
I20260812 11:25:03.134815 28762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "266898dd7ac94e7dbf45c3df03e35b30"
format_stamp: "Formatted at 2026-08-12 11:25:03 on dist-test-slave-h6n0"
server_key: "abb3b5c392621372840ed9032a62e782"
server_key_iv: "596f9a67154961f5a3229f8f554b69cb"
server_key_version: "encryptionkey@0"
I20260812 11:25:03.139870 28762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:25:03.143289 28782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:03.144680 28762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:25:03.144940 28762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "266898dd7ac94e7dbf45c3df03e35b30"
format_stamp: "Formatted at 2026-08-12 11:25:03 on dist-test-slave-h6n0"
server_key: "abb3b5c392621372840ed9032a62e782"
server_key_iv: "596f9a67154961f5a3229f8f554b69cb"
server_key_version: "encryptionkey@0"
I20260812 11:25:03.145167 28762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:03.162786 28762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:03.163738 28762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:03.164049 28762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:03.165169 28762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:03.166816 28762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:03.166931 28762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:03.167070 28762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:03.167188 28762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:03.204624 28762 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:33771
I20260812 11:25:03.204681 28971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:33771 every 8 connection(s)
I20260812 11:25:03.205920 28762 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:25:03.214885 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 28762
I20260812 11:25:03.215080 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:25:03.215559 12731 external_mini_cluster.cc:1428] Setting key 81999fe9b8483958ae24f3290048cda8
I20260812 11:25:03.219331 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40667
--builtin_ntp_servers=127.12.110.212:45997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260812 11:25:03.219604 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:03.208595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:43359 (local address 127.12.110.254:40667)
0812 11:25:03.208770 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:03.208775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:03.210009 (+ 1234us) server_negotiation.cc:407] Connection header received
0812 11:25:03.211868 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:03.211874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:03.211981 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:03.212110 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:03.213140 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.213873 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.215658 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.216079 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.216400 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:03.218423 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:03.218428 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:03.218431 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:03.218449 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:03.218488 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:03.218501 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:03.218654 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:03.218950 (+ 296us) server_negotiation.cc:299] Negotiation successful
0812 11:25:03.219036 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:25:03.220688 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:03.209005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40667 (local address 127.12.110.194:43359)
0812 11:25:03.209851 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:03.209896 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:03.211595 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:03.212338 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:03.212345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:03.212424 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:03.212964 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:03.212980 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.214039 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.214052 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:03.215489 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:03.215502 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.216548 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.216559 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:03.216963 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:03.218009 (+ 1046us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:25:03.218012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:03.218082 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:25:03.218188 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:03.218676 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:03.218687 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:03.218695 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:25:03.218893 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 11:25:03.219322 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":272,"threads_started":1}
I20260812 11:25:03.222133 28972 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40667
I20260812 11:25:03.222473 28972 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:03.223652 28972 heartbeater.cc:511] Master 127.12.110.254:40667 requested a full tablet report, sending...
I20260812 11:25:03.225503 28412 ts_manager.cc:194] Registered new tserver with Master: 266898dd7ac94e7dbf45c3df03e35b30 (127.12.110.194:33771)
I20260812 11:25:03.226707 28412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.194:43359
W20260812 11:25:03.569069 28980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:03.569329 28980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:03.569473 28980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:03.576850 28980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:03.576946 28980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:03.577104 28980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:25:03.587095 28980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:03.588604 28980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:03.590677 28980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:03.604382 28992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:03.604287 28994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:03.604344 28999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:03.605234 28980 server_base.cc:1034] running on GCE node
I20260812 11:25:03.605827 28980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:03.606787 28980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:03.608033 28980 hybrid_clock.cc:648] HybridClock initialized: now 1786533903607965 us; error 54 us; skew 500 ppm
I20260812 11:25:03.608362 28980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:03.610862 28980 webserver.cc:460] Webserver started at http://127.12.110.195:43337/ using document root <none> and password file <none>
I20260812 11:25:03.611552 28980 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:03.611686 28980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:03.612025 28980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:03.614822 28980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "726506e8a2154a9aaf9ddffbfeb9cdb5"
format_stamp: "Formatted at 2026-08-12 11:25:03 on dist-test-slave-h6n0"
server_key: "4a8f3824805d9182e0646faea296845a"
server_key_iv: "6e9e360d0c1b4cbd3c84a58279c94663"
server_key_version: "encryptionkey@0"
I20260812 11:25:03.615638 28980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "726506e8a2154a9aaf9ddffbfeb9cdb5"
format_stamp: "Formatted at 2026-08-12 11:25:03 on dist-test-slave-h6n0"
server_key: "4a8f3824805d9182e0646faea296845a"
server_key_iv: "6e9e360d0c1b4cbd3c84a58279c94663"
server_key_version: "encryptionkey@0"
I20260812 11:25:03.620577 28980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:25:03.623980 29006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:03.625188 28980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 11:25:03.625371 28980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "726506e8a2154a9aaf9ddffbfeb9cdb5"
format_stamp: "Formatted at 2026-08-12 11:25:03 on dist-test-slave-h6n0"
server_key: "4a8f3824805d9182e0646faea296845a"
server_key_iv: "6e9e360d0c1b4cbd3c84a58279c94663"
server_key_version: "encryptionkey@0"
I20260812 11:25:03.625589 28980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:03.645197 28980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:03.646163 28980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:03.646513 28980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:03.647578 28980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:03.649537 28980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:03.649677 28980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:03.649917 28980 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:03.650082 28980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:03.686379 28980 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:44745
I20260812 11:25:03.686448 29200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:44745 every 8 connection(s)
I20260812 11:25:03.687690 28980 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:25:03.692232 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 28980
I20260812 11:25:03.692456 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:25:03.693049 12731 external_mini_cluster.cc:1428] Setting key 60a5120eaa77bba8ca4e458488bcae70
I20260812 11:25:03.701443 28756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:03.690318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:57461 (local address 127.12.110.254:40667)
0812 11:25:03.690604 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:03.690611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:03.692415 (+ 1804us) server_negotiation.cc:407] Connection header received
0812 11:25:03.694874 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:03.694878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:03.694947 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:03.695085 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:03.696273 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.697007 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.698600 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.699013 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.699115 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:03.700706 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:03.700710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:03.700714 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:03.700730 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:03.700771 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:03.700790 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:03.700871 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:03.701097 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 11:25:03.701179 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:25:03.702374 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:03.690954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40667 (local address 127.12.110.195:57461)
0812 11:25:03.692127 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:03.692199 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:03.694509 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:03.695340 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:03.695354 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:03.695463 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:03.696090 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:03.696117 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.697161 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.697165 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:03.698404 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:03.698416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.699299 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.699305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:03.699503 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:03.700398 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:25:03.700401 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:03.700437 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:25:03.700510 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:03.700892 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:03.700900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:03.700906 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:25:03.701047 (+ 141us) client_negotiation.cc:240] Negotiation successful
0812 11:25:03.701328 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":338,"threads_started":1}
I20260812 11:25:03.703701 29202 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40667
I20260812 11:25:03.703990 29202 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:03.704715 29202 heartbeater.cc:511] Master 127.12.110.254:40667 requested a full tablet report, sending...
I20260812 11:25:03.706092 28412 ts_manager.cc:194] Registered new tserver with Master: 726506e8a2154a9aaf9ddffbfeb9cdb5 (127.12.110.195:44745)
I20260812 11:25:03.707048 28412 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.195:57461
I20260812 11:25:03.709131 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:25:03.715682 28756 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:25:03.713874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.12.110.254:40667)
0812 11:25:03.714082 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:03.714089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:03.714681 (+ 592us) server_negotiation.cc:407] Connection header received
0812 11:25:03.714826 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:03.714830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:03.714891 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:03.714976 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:03.715472 (+ 496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34074: BlockingRecv error: recv got EOF from 127.0.0.1:34074 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:25:03.716706 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 28517
I20260812 11:25:03.727458 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:03.719751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.12.110.193:34663)
0812 11:25:03.720361 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:03.720367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:03.720395 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:25:03.720447 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:03.720452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:03.720629 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:03.720804 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:03.721255 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.722750 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.724154 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.724700 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.724852 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:03.726021 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:03.726029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:03.726042 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:03.726615 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:03.726661 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:03.726680 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:03.726812 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:03.726982 (+ 170us) server_negotiation.cc:299] Negotiation successful
0812 11:25:03.727117 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":326,"threads_started":1}
I20260812 11:25:03.728787 28741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:03.810714 28751 heartbeater.cc:503] Master 127.12.110.254:40667 was elected leader, sending a full tablet report...
I20260812 11:25:03.812040 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 28517
W20260812 11:25:03.836630 28386 connection.cc:535] server connection from 127.12.110.193:60769 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:25:03.836903 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 28762
I20260812 11:25:03.845641 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:03.838068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.12.110.194:33771)
0812 11:25:03.838644 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:03.838654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:03.838691 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:25:03.838763 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:03.838769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:03.838993 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:03.839126 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:03.839505 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.840495 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.841648 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.842173 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.842271 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:03.843507 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:03.843516 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:03.843529 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:03.844107 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:03.844159 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:03.844178 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:03.844272 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:03.844689 (+ 417us) server_negotiation.cc:299] Negotiation successful
0812 11:25:03.845124 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":304,"threads_started":1}
I20260812 11:25:03.847601 28961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:03.931540 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 28762
I20260812 11:25:03.955773 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 28980
I20260812 11:25:03.964289 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:03.956834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.12.110.195:44745)
0812 11:25:03.957469 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:03.957476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:03.957505 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:25:03.957569 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:03.957575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:03.957787 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:03.957958 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:03.958424 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.959562 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.960816 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:03.961370 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:03.961515 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:03.962844 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:03.962853 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:03.962865 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:03.963430 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:03.963473 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:03.963491 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:03.963622 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:03.963790 (+ 168us) server_negotiation.cc:299] Negotiation successful
0812 11:25:03.963929 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":350,"threads_started":1}
I20260812 11:25:03.965955 29195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:04.048403 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 28980
I20260812 11:25:04.073429 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 28346
I20260812 11:25:04.074792 28465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:04.183279 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 28346
2026-08-12T11:25:04Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2607 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-12T11:25:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:25:04Z Disabled control of system clock
I20260812 11:25:04.286002 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40141
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:40141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 11:25:04.631301 29235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:04.631594 29235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:04.631752 29235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:04.638906 29235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:25:04.639015 29235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:04.639106 29235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:04.639243 29235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:25:04.639326 29235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:25:04.648520 29235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:40141
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:04.649766 29235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:04.651531 29235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:04.663623 29251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:04.663740 29246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:04.663806 29245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:04.665182 29235 server_base.cc:1034] running on GCE node
I20260812 11:25:04.666024 29235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:04.667517 29235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:04.668740 29235 hybrid_clock.cc:648] HybridClock initialized: now 1786533904668686 us; error 45 us; skew 500 ppm
I20260812 11:25:04.669128 29235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:04.671613 29235 webserver.cc:460] Webserver started at http://127.12.110.254:38309/ using document root <none> and password file <none>
I20260812 11:25:04.672325 29235 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:04.672427 29235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:04.672747 29235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:04.676599 29235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "9887e46f6b094e9f824c464060890b9c"
format_stamp: "Formatted at 2026-08-12 11:25:04 on dist-test-slave-h6n0"
server_key: "2df3f2618806667a1cb2df6b5c40cd1b"
server_key_iv: "a415060adeb88cce995d8206d049335e"
server_key_version: "encryptionkey@0"
I20260812 11:25:04.677452 29235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "9887e46f6b094e9f824c464060890b9c"
format_stamp: "Formatted at 2026-08-12 11:25:04 on dist-test-slave-h6n0"
server_key: "2df3f2618806667a1cb2df6b5c40cd1b"
server_key_iv: "a415060adeb88cce995d8206d049335e"
server_key_version: "encryptionkey@0"
I20260812 11:25:04.682416 29235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 11:25:04.685706 29257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:04.687117 29235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:25:04.687309 29235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "9887e46f6b094e9f824c464060890b9c"
format_stamp: "Formatted at 2026-08-12 11:25:04 on dist-test-slave-h6n0"
server_key: "2df3f2618806667a1cb2df6b5c40cd1b"
server_key_iv: "a415060adeb88cce995d8206d049335e"
server_key_version: "encryptionkey@0"
I20260812 11:25:04.687558 29235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:04.693706 29235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:04.694471 29235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:04.694803 29235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:04.716017 29235 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:40141
I20260812 11:25:04.716048 29351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:40141 every 8 connection(s)
I20260812 11:25:04.717438 29235 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:25:04.721958 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:25:04.726878 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 29235
I20260812 11:25:04.727141 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:25:04.727802 12731 external_mini_cluster.cc:1428] Setting key 07d9d84ba22c4c503698f541766ae731
I20260812 11:25:04.733291 29352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c: Bootstrap starting.
I20260812 11:25:04.736821 29358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:04.730095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.12.110.254:40141)
0812 11:25:04.731321 (+ 1226us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:04.731332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:04.731385 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:25:04.733077 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:04.733100 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:04.733566 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:04.733899 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:04.735173 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:04.735180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:04.735194 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:04.735210 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:04.735288 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:04.735306 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:04.735490 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:04.735664 (+ 174us) server_negotiation.cc:299] Negotiation successful
0812 11:25:04.735972 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":495,"threads_started":1}
I20260812 11:25:04.737227 29352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c: Neither blocks nor log segments found. Creating new log.
I20260812 11:25:04.738567 29352 log.cc:824] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:25:04.741549 29352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c: No bootstrap required, opened a new log
I20260812 11:25:04.749233 29352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9887e46f6b094e9f824c464060890b9c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40141 } }
I20260812 11:25:04.749579 29352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:25:04.749712 29352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9887e46f6b094e9f824c464060890b9c, State: Initialized, Role: FOLLOWER
I20260812 11:25:04.750277 29352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9887e46f6b094e9f824c464060890b9c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40141 } }
I20260812 11:25:04.750511 29352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:25:04.750676 29352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:25:04.750859 29352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:25:04.752992 29352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9887e46f6b094e9f824c464060890b9c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40141 } }
I20260812 11:25:04.753479 29352 leader_election.cc:302] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9887e46f6b094e9f824c464060890b9c; no voters:
I20260812 11:25:04.754127 29352 leader_election.cc:288] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:25:04.754311 29363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:25:04.755425 29363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [term 1 LEADER]: Becoming Leader. State: Replica: 9887e46f6b094e9f824c464060890b9c, State: Running, Role: LEADER
I20260812 11:25:04.755964 29363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9887e46f6b094e9f824c464060890b9c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40141 } }
I20260812 11:25:04.756520 29352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:25:04.758742 29365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9887e46f6b094e9f824c464060890b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9887e46f6b094e9f824c464060890b9c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40141 } } }
I20260812 11:25:04.758824 29367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9887e46f6b094e9f824c464060890b9c. Latest consensus state: current_term: 1 leader_uuid: "9887e46f6b094e9f824c464060890b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9887e46f6b094e9f824c464060890b9c" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40141 } } }
I20260812 11:25:04.759179 29365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [sys.catalog]: This master's current role is: LEADER
I20260812 11:25:04.759243 29367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c [sys.catalog]: This master's current role is: LEADER
I20260812 11:25:04.760215 29374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:25:04.765888 29374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:25:04.780458 29374 catalog_manager.cc:1347] Generated new cluster ID: 7b17e618f0744a67a9dcb32845f16cdf
I20260812 11:25:04.780606 29374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:25:04.808759 29374 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:25:04.809291 29374 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:25:04.819058 29374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9887e46f6b094e9f824c464060890b9c: Generated new TSK 0
I20260812 11:25:04.820088 29374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:25:04.833274 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40141
--builtin_ntp_servers=127.12.110.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260812 11:25:05.198410 29400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:05.198846 29400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:05.199023 29400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:05.206378 29400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:05.206509 29400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:05.206715 29400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:25:05.215755 29400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:05.217065 29400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:05.218823 29400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:05.232144 29410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:05.232167 29411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:05.232125 29414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:05.236284 29400 server_base.cc:1034] running on GCE node
I20260812 11:25:05.236999 29400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:05.237972 29400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:05.239212 29400 hybrid_clock.cc:648] HybridClock initialized: now 1786533905239145 us; error 52 us; skew 500 ppm
I20260812 11:25:05.239531 29400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:05.245187 29400 webserver.cc:460] Webserver started at http://127.12.110.193:40181/ using document root <none> and password file <none>
I20260812 11:25:05.245904 29400 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:05.246059 29400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:05.246400 29400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:05.249310 29400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "c64a9831fd074855941296b1774e6341"
format_stamp: "Formatted at 2026-08-12 11:25:05 on dist-test-slave-h6n0"
server_key: "930751aa35561224cb6d8c0b9cc5468e"
server_key_iv: "1973fb2c50c8db97d7f9cb81223d8511"
server_key_version: "encryptionkey@0"
I20260812 11:25:05.250144 29400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "c64a9831fd074855941296b1774e6341"
format_stamp: "Formatted at 2026-08-12 11:25:05 on dist-test-slave-h6n0"
server_key: "930751aa35561224cb6d8c0b9cc5468e"
server_key_iv: "1973fb2c50c8db97d7f9cb81223d8511"
server_key_version: "encryptionkey@0"
I20260812 11:25:05.256708 29400 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 11:25:05.261581 29427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:05.263396 29400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260812 11:25:05.263636 29400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "c64a9831fd074855941296b1774e6341"
format_stamp: "Formatted at 2026-08-12 11:25:05 on dist-test-slave-h6n0"
server_key: "930751aa35561224cb6d8c0b9cc5468e"
server_key_iv: "1973fb2c50c8db97d7f9cb81223d8511"
server_key_version: "encryptionkey@0"
I20260812 11:25:05.263898 29400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:05.270682 29400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:05.271667 29400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:05.271986 29400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:05.273026 29400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:05.274684 29400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:05.274822 29400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:05.274976 29400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:05.275096 29400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:05.315085 29400 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:44409
I20260812 11:25:05.315131 29622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:44409 every 8 connection(s)
I20260812 11:25:05.316442 29400 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:25:05.320408 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 29400
I20260812 11:25:05.320642 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:25:05.321425 12731 external_mini_cluster.cc:1428] Setting key b92d7b801f7c380ee147a621b6ef6ca4
I20260812 11:25:05.324259 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40141
--builtin_ntp_servers=127.12.110.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 11:25:05.325527 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:05.319317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:60097 (local address 127.12.110.254:40141)
0812 11:25:05.319840 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:05.319849 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:05.321055 (+ 1206us) server_negotiation.cc:407] Connection header received
0812 11:25:05.323015 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:05.323021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:05.323087 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:05.323224 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:05.324716 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:05.324721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:05.324726 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:05.324751 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:05.324794 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:05.324806 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:05.324948 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:05.325152 (+ 204us) server_negotiation.cc:299] Negotiation successful
0812 11:25:05.325220 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":286,"threads_started":1}
I20260812 11:25:05.326382 29626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:05.319797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40141 (local address 127.12.110.193:60097)
0812 11:25:05.320892 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:05.320939 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:05.322751 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:05.323452 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:05.323459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:05.323539 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:05.324364 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:25:05.324369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:05.324430 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:25:05.324550 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:05.324915 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:05.324929 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:05.324937 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:25:05.325230 (+ 293us) client_negotiation.cc:240] Negotiation successful
0812 11:25:05.325466 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":325,"threads_started":1}
I20260812 11:25:05.327939 29623 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40141
I20260812 11:25:05.328286 29623 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:05.328956 29623 heartbeater.cc:511] Master 127.12.110.254:40141 requested a full tablet report, sending...
I20260812 11:25:05.331459 29288 ts_manager.cc:194] Registered new tserver with Master: c64a9831fd074855941296b1774e6341 (127.12.110.193:44409)
W20260812 11:25:05.668668 29627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:05.668967 29627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:05.669133 29627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:05.676160 29627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:05.676290 29627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:05.676470 29627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:25:05.685114 29627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:05.686354 29627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:05.688122 29627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:05.701360 29635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:05.701305 29637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:05.701402 29640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:05.702304 29627 server_base.cc:1034] running on GCE node
I20260812 11:25:05.703070 29627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:05.704002 29627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:05.705228 29627 hybrid_clock.cc:648] HybridClock initialized: now 1786533905705168 us; error 46 us; skew 500 ppm
I20260812 11:25:05.705530 29627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:05.708021 29627 webserver.cc:460] Webserver started at http://127.12.110.194:33377/ using document root <none> and password file <none>
I20260812 11:25:05.708693 29627 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:05.708796 29627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:05.709120 29627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:05.712050 29627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "076687f7f3cb495d9ddb33cc66abf019"
format_stamp: "Formatted at 2026-08-12 11:25:05 on dist-test-slave-h6n0"
server_key: "854a980ca976e1c36e84bf866db09732"
server_key_iv: "39b99479e08e15f311838eb167e8cbb7"
server_key_version: "encryptionkey@0"
I20260812 11:25:05.712847 29627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "076687f7f3cb495d9ddb33cc66abf019"
format_stamp: "Formatted at 2026-08-12 11:25:05 on dist-test-slave-h6n0"
server_key: "854a980ca976e1c36e84bf866db09732"
server_key_iv: "39b99479e08e15f311838eb167e8cbb7"
server_key_version: "encryptionkey@0"
I20260812 11:25:05.718164 29627 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:25:05.721786 29650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:05.723225 29627 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:25:05.723440 29627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "076687f7f3cb495d9ddb33cc66abf019"
format_stamp: "Formatted at 2026-08-12 11:25:05 on dist-test-slave-h6n0"
server_key: "854a980ca976e1c36e84bf866db09732"
server_key_iv: "39b99479e08e15f311838eb167e8cbb7"
server_key_version: "encryptionkey@0"
I20260812 11:25:05.723683 29627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:05.729817 29627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:05.730700 29627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:05.731038 29627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:05.732357 29627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:05.734614 29627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:05.734741 29627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 11:25:05.734898 29627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:05.735005 29627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:05.775102 29627 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:34813
I20260812 11:25:05.775162 29861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:34813 every 8 connection(s)
I20260812 11:25:05.776417 29627 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:25:05.777858 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 29627
I20260812 11:25:05.778295 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:25:05.779024 12731 external_mini_cluster.cc:1428] Setting key af60b226835ccbe944ae95ac479abd18
I20260812 11:25:05.781915 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40141
--builtin_ntp_servers=127.12.110.212:40763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260812 11:25:05.786427 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:05.780031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:42073 (local address 127.12.110.254:40141)
0812 11:25:05.780249 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:05.780256 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:05.781918 (+ 1662us) server_negotiation.cc:407] Connection header received
0812 11:25:05.783952 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:05.783957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:05.784038 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:05.784171 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:05.785617 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:05.785623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:05.785628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:05.785648 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:05.785691 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:05.785703 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:05.785857 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:05.786077 (+ 220us) server_negotiation.cc:299] Negotiation successful
0812 11:25:05.786153 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260812 11:25:05.787382 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:05.780579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40141 (local address 127.12.110.194:42073)
0812 11:25:05.781717 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:05.781765 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:05.783590 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:05.784430 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:05.784438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:05.784544 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:05.785297 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:25:05.785300 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:05.785345 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:25:05.785430 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:05.785823 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:05.785836 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:05.785844 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:25:05.786146 (+ 302us) client_negotiation.cc:240] Negotiation successful
0812 11:25:05.786389 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":400,"threads_started":1}
I20260812 11:25:05.791627 29863 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40141
I20260812 11:25:05.791985 29863 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:05.792740 29863 heartbeater.cc:511] Master 127.12.110.254:40141 requested a full tablet report, sending...
I20260812 11:25:05.794332 29288 ts_manager.cc:194] Registered new tserver with Master: 076687f7f3cb495d9ddb33cc66abf019 (127.12.110.194:34813)
W20260812 11:25:06.122644 29866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:06.122920 29866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:06.123093 29866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:06.130510 29866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:06.130621 29866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:06.130831 29866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:25:06.139550 29866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40141
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:06.140774 29866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:06.142551 29866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:06.155630 29872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:06.155651 29874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:06.156066 29879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:06.156368 29866 server_base.cc:1034] running on GCE node
I20260812 11:25:06.157033 29866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:06.157944 29866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:06.159196 29866 hybrid_clock.cc:648] HybridClock initialized: now 1786533906159170 us; error 43 us; skew 500 ppm
I20260812 11:25:06.159513 29866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:06.162274 29866 webserver.cc:460] Webserver started at http://127.12.110.195:40541/ using document root <none> and password file <none>
I20260812 11:25:06.163245 29866 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:06.163373 29866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:06.163784 29866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:06.166754 29866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "481559fc35d547ab8528eb808c45f55c"
format_stamp: "Formatted at 2026-08-12 11:25:06 on dist-test-slave-h6n0"
server_key: "57ddacf9aa38bc4efa434c7242e8cbd0"
server_key_iv: "1d33de0242af23e7e3647274d026b13f"
server_key_version: "encryptionkey@0"
I20260812 11:25:06.167626 29866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "481559fc35d547ab8528eb808c45f55c"
format_stamp: "Formatted at 2026-08-12 11:25:06 on dist-test-slave-h6n0"
server_key: "57ddacf9aa38bc4efa434c7242e8cbd0"
server_key_iv: "1d33de0242af23e7e3647274d026b13f"
server_key_version: "encryptionkey@0"
I20260812 11:25:06.172564 29866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:25:06.175804 29894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:06.177016 29866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:25:06.177196 29866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "481559fc35d547ab8528eb808c45f55c"
format_stamp: "Formatted at 2026-08-12 11:25:06 on dist-test-slave-h6n0"
server_key: "57ddacf9aa38bc4efa434c7242e8cbd0"
server_key_iv: "1d33de0242af23e7e3647274d026b13f"
server_key_version: "encryptionkey@0"
I20260812 11:25:06.177425 29866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:06.183176 29866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:06.183928 29866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:06.184211 29866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:06.185299 29866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:06.186925 29866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:06.187071 29866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:06.187206 29866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:06.187300 29866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:06.223865 29866 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:37727
I20260812 11:25:06.224020 30101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:37727 every 8 connection(s)
I20260812 11:25:06.225193 29866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:25:06.233592 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:06.228035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:57965 (local address 127.12.110.254:40141)
0812 11:25:06.228289 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:06.228296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:06.229652 (+ 1356us) server_negotiation.cc:407] Connection header received
0812 11:25:06.231395 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:06.231401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:06.231478 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:06.231607 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:06.232807 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:06.232812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:06.232817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:06.232834 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:06.232879 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:06.232890 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:06.233023 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:06.233219 (+ 196us) server_negotiation.cc:299] Negotiation successful
0812 11:25:06.233291 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260812 11:25:06.234341 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 29866
I20260812 11:25:06.234479 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:06.228615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40141 (local address 127.12.110.195:57965)
0812 11:25:06.229484 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:06.229527 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:06.231136 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:06.231820 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:06.231827 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:06.231905 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:06.232532 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:25:06.232535 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:06.232569 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:25:06.232641 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:06.232995 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:06.233006 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:06.233013 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:25:06.233298 (+ 285us) client_negotiation.cc:240] Negotiation successful
0812 11:25:06.233520 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":272,"threads_started":1}
I20260812 11:25:06.234593 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:25:06.235178 12731 external_mini_cluster.cc:1428] Setting key 7df786d380129664d069665868c2e1fa
I20260812 11:25:06.235847 30103 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40141
I20260812 11:25:06.236161 30103 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:06.236918 30103 heartbeater.cc:511] Master 127.12.110.254:40141 requested a full tablet report, sending...
I20260812 11:25:06.238330 29288 ts_manager.cc:194] Registered new tserver with Master: 481559fc35d547ab8528eb808c45f55c (127.12.110.195:37727)
I20260812 11:25:06.250378 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260812 11:25:06.256640 29625 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:25:06.254948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.12.110.254:40141)
0812 11:25:06.255142 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:06.255149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:06.255755 (+ 606us) server_negotiation.cc:407] Connection header received
0812 11:25:06.255868 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:06.255872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:06.255919 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:06.256000 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:06.256406 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36612: BlockingRecv error: recv got EOF from 127.0.0.1:36612 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:25:06.257356 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 29400
I20260812 11:25:06.261111 30120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:06.258562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.12.110.193:44409)
0812 11:25:06.259085 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:06.259091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:06.259116 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:25:06.259170 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:06.259175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:06.259354 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:06.259473 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:06.260439 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:06.260446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:06.260456 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:06.260469 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:06.260515 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:06.260534 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:06.260634 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:06.260713 (+ 79us) server_negotiation.cc:299] Negotiation successful
0812 11:25:06.260819 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260812 11:25:06.262298 29616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:06.347160 29623 heartbeater.cc:503] Master 127.12.110.254:40141 was elected leader, sending a full tablet report...
I20260812 11:25:06.348606 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 29400
W20260812 11:25:06.371212 29262 connection.cc:535] server connection from 127.12.110.193:60097 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:25:06.371591 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 29627
I20260812 11:25:06.375587 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:06.372555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.12.110.194:34813)
0812 11:25:06.373022 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:06.373030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:06.373055 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:25:06.373120 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:06.373126 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:06.373350 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:06.373543 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:06.374573 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:06.374582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:06.374602 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:06.374617 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:06.374670 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:06.374689 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:06.374836 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:06.374963 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 11:25:06.375071 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":211,"threads_started":1}
I20260812 11:25:06.376922 29856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:06.458658 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 29627
I20260812 11:25:06.486260 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 29866
I20260812 11:25:06.490674 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:06.487407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.12.110.195:37727)
0812 11:25:06.488109 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:06.488117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:06.488148 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:25:06.488212 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:06.488219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:06.488444 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:06.488624 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:06.489841 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:06.489848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:06.489858 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:25:06.489870 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:06.489913 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:25:06.489926 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:06.490127 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:06.490199 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 11:25:06.490310 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":369,"threads_started":1}
I20260812 11:25:06.492048 30088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:06.576460 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 29866
I20260812 11:25:06.603444 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 29235
I20260812 11:25:06.604828 29341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:06.709283 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 29235
2026-08-12T11:25:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2528 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:25:06 dist-test-slave-h6n0 krb5kdc[30140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:25:06 dist-test-slave-h6n0 krb5kdc[30140](info): set up 2 sockets
Aug 12 11:25:06 dist-test-slave-h6n0 krb5kdc[30140](info): commencing operation
krb5kdc: starting...
W20260812 11:25:08.092060 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:25:08 dist-test-slave-h6n0 krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:25:08.436996 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40469
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:40469
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:25:08.785210 30163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:08.785576 30163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:08.785758 30163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:08.793015 30163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:25:08.793186 30163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:08.793308 30163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:08.793422 30163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:25:08.793515 30163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:25:08.803690 30163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:40469
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:08.805109 30163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:08.806411 30163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:08.819681 30175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:08.819754 30178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:08.819850 30181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:08.820775 30163 server_base.cc:1034] running on GCE node
I20260812 11:25:08.821553 30163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:08.823012 30163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:08.824288 30163 hybrid_clock.cc:648] HybridClock initialized: now 1786533908824253 us; error 49 us; skew 500 ppm
I20260812 11:25:08.824746 30163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:25:08 dist-test-slave-h6n0 krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533908, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:25:08.828164 30163 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:25:08.829767 30163 webserver.cc:460] Webserver started at http://127.12.110.254:40491/ using document root <none> and password file <none>
I20260812 11:25:08.830447 30163 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:08.830626 30163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:08.830952 30163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:08.833977 30163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "edd4dca6da4e49edbcf78a3453a9d401"
format_stamp: "Formatted at 2026-08-12 11:25:08 on dist-test-slave-h6n0"
server_key: "db1bb05de113f9c6fe12d511f598b11b"
server_key_iv: "68785c495ae2336950376044530351d9"
server_key_version: "encryptionkey@0"
I20260812 11:25:08.834834 30163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "edd4dca6da4e49edbcf78a3453a9d401"
format_stamp: "Formatted at 2026-08-12 11:25:08 on dist-test-slave-h6n0"
server_key: "db1bb05de113f9c6fe12d511f598b11b"
server_key_iv: "68785c495ae2336950376044530351d9"
server_key_version: "encryptionkey@0"
I20260812 11:25:08.840449 30163 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:25:08.844168 30189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:08.845574 30163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:25:08.845788 30163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "edd4dca6da4e49edbcf78a3453a9d401"
format_stamp: "Formatted at 2026-08-12 11:25:08 on dist-test-slave-h6n0"
server_key: "db1bb05de113f9c6fe12d511f598b11b"
server_key_iv: "68785c495ae2336950376044530351d9"
server_key_version: "encryptionkey@0"
I20260812 11:25:08.846009 30163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:08.870922 30163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:08.875538 30163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:08.875846 30163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:08.896049 30163 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:40469
I20260812 11:25:08.896103 30280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:40469 every 8 connection(s)
I20260812 11:25:08.897444 30163 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:25:08.900477 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 30163
I20260812 11:25:08.900724 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:25:08.901582 12731 external_mini_cluster.cc:1428] Setting key f1319a77cb39d3ecd438ff3bdfb29b31
I20260812 11:25:08.901789 30281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:25:08.916833 30281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401: Bootstrap starting.
Aug 12 11:25:08 dist-test-slave-h6n0 krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:25:08.920704 30281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401: Neither blocks nor log segments found. Creating new log.
I20260812 11:25:08.922122 30281 log.cc:824] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401: Log is configured to *not* fsync() on all Append() calls
I20260812 11:25:08.925751 30281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401: No bootstrap required, opened a new log
I20260812 11:25:08.927503 30289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:08.903673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.12.110.254:40469)
0812 11:25:08.904448 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:08.904460 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:08.904506 (+ 46us) server_negotiation.cc:407] Connection header received
0812 11:25:08.906510 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:08.906586 (+ 76us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:08.906962 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:08.907272 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:08.909118 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:08.910422 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:08.912309 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:08.913261 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:08.913421 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:08.918407 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:08.918421 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:08.918437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:08.918506 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:08.922580 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:08.923515 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:08.923522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:08.923527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:08.923751 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:08.924217 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:08.924222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:08.924225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:08.924962 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:08.925296 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:08.925688 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 11:25:08.926115 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":352,"threads_started":1}
I20260812 11:25:08.934249 30281 raft_consensus.cc:348] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4dca6da4e49edbcf78a3453a9d401" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40469 } }
I20260812 11:25:08.934650 30281 raft_consensus.cc:374] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:25:08.934741 30281 raft_consensus.cc:729] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd4dca6da4e49edbcf78a3453a9d401, State: Initialized, Role: FOLLOWER
I20260812 11:25:08.935325 30281 consensus_queue.cc:260] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4dca6da4e49edbcf78a3453a9d401" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40469 } }
I20260812 11:25:08.935537 30281 raft_consensus.cc:388] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:25:08.935637 30281 raft_consensus.cc:482] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:25:08.935757 30281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:25:08.937808 30281 raft_consensus.cc:504] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4dca6da4e49edbcf78a3453a9d401" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40469 } }
I20260812 11:25:08.938238 30281 leader_election.cc:302] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edd4dca6da4e49edbcf78a3453a9d401; no voters:
I20260812 11:25:08.938869 30281 leader_election.cc:288] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:25:08.939025 30291 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:25:08.939725 30291 raft_consensus.cc:686] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [term 1 LEADER]: Becoming Leader. State: Replica: edd4dca6da4e49edbcf78a3453a9d401, State: Running, Role: LEADER
I20260812 11:25:08.940282 30291 consensus_queue.cc:237] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4dca6da4e49edbcf78a3453a9d401" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40469 } }
I20260812 11:25:08.940960 30281 sys_catalog.cc:564] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:25:08.942965 30297 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [sys.catalog]: SysCatalogTable state changed. Reason: New leader edd4dca6da4e49edbcf78a3453a9d401. Latest consensus state: current_term: 1 leader_uuid: "edd4dca6da4e49edbcf78a3453a9d401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4dca6da4e49edbcf78a3453a9d401" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40469 } } }
I20260812 11:25:08.943341 30297 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [sys.catalog]: This master's current role is: LEADER
I20260812 11:25:08.944123 30296 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edd4dca6da4e49edbcf78a3453a9d401" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd4dca6da4e49edbcf78a3453a9d401" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40469 } } }
I20260812 11:25:08.944366 30308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:25:08.945425 30296 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401 [sys.catalog]: This master's current role is: LEADER
I20260812 11:25:08.950875 30308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:25:08.960783 30308 catalog_manager.cc:1347] Generated new cluster ID: 9555015e10b9486ea2e877381ad3421b
I20260812 11:25:08.960918 30308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:25:08.977365 30308 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:25:08.978626 30308 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:25:08.987952 30308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401: Generated new TSK 0
I20260812 11:25:08.988747 30308 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:25:09.078948 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40469
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:25:09.439471 30335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:09.439755 30335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:09.439851 30335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:09.447723 30335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:09.447834 30335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:09.447952 30335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:25:09.458359 30335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:09.459687 30335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:09.460951 30335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:09.474493 30345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:09.474644 30344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:09.474583 30348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:09.475603 30335 server_base.cc:1034] running on GCE node
I20260812 11:25:09.476164 30335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:09.477020 30335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:09.478217 30335 hybrid_clock.cc:648] HybridClock initialized: now 1786533909478160 us; error 49 us; skew 500 ppm
I20260812 11:25:09.478564 30335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:25:09 dist-test-slave-h6n0 krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:25:09.481863 30335 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:25:09.483299 30335 webserver.cc:460] Webserver started at http://127.12.110.193:46443/ using document root <none> and password file <none>
I20260812 11:25:09.483992 30335 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:09.484102 30335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:09.484402 30335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:09.487517 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "c62c697c53f34b64b2c71a90e1b74d10"
format_stamp: "Formatted at 2026-08-12 11:25:09 on dist-test-slave-h6n0"
server_key: "2df352cca0fcffe5ca72a8884886d245"
server_key_iv: "2573257e05ee9cd46b07c5e4f9f2951d"
server_key_version: "encryptionkey@0"
I20260812 11:25:09.488365 30335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "c62c697c53f34b64b2c71a90e1b74d10"
format_stamp: "Formatted at 2026-08-12 11:25:09 on dist-test-slave-h6n0"
server_key: "2df352cca0fcffe5ca72a8884886d245"
server_key_iv: "2573257e05ee9cd46b07c5e4f9f2951d"
server_key_version: "encryptionkey@0"
I20260812 11:25:09.493567 30335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:25:09.498483 30362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:09.500139 30335 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:25:09.500286 30335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "c62c697c53f34b64b2c71a90e1b74d10"
format_stamp: "Formatted at 2026-08-12 11:25:09 on dist-test-slave-h6n0"
server_key: "2df352cca0fcffe5ca72a8884886d245"
server_key_iv: "2573257e05ee9cd46b07c5e4f9f2951d"
server_key_version: "encryptionkey@0"
I20260812 11:25:09.500538 30335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:09.524061 30335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:09.528213 30335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:09.528520 30335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:09.529512 30335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:09.531031 30335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:09.531147 30335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:09.531332 30335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:09.531409 30335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:09.567945 30335 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:39555
I20260812 11:25:09.568048 30577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:39555 every 8 connection(s)
I20260812 11:25:09.569271 30335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:25:09.573751 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 30335
I20260812 11:25:09.574002 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:25:09.574733 12731 external_mini_cluster.cc:1428] Setting key 07d978e68ad6d5cfe05882a262acf86f
Aug 12 11:25:09 dist-test-slave-h6n0 krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533909, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:25:09.590240 30580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:09.571950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:56797 (local address 127.12.110.254:40469)
0812 11:25:09.572467 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:09.572472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:09.573537 (+ 1065us) server_negotiation.cc:407] Connection header received
0812 11:25:09.575932 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:09.575939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:09.576048 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:09.576177 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:09.577671 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:09.578396 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:09.579963 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:09.580495 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:09.580667 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:09.584597 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:09.584602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:09.584606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:09.584638 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:09.587232 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:09.587968 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:09.587973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:09.587976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:09.588109 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:09.588549 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:09.588553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:09.588555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:09.588806 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:09.588965 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:09.589648 (+ 683us) server_negotiation.cc:299] Negotiation successful
0812 11:25:09.589866 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":279,"threads_started":1}
I20260812 11:25:09.591223 30581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:09.572298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40469 (local address 127.12.110.193:56797)
0812 11:25:09.573231 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:09.573309 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:09.575620 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:09.576426 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:09.576434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:09.576945 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:09.577492 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:09.577514 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:09.578591 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:09.578595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:09.579801 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:09.579813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:09.580757 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:09.580768 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:09.581034 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:09.581794 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:25:09.581815 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:09.584369 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:09.587373 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:25:09.587382 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:25:09.587397 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:25:09.587822 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:25:09.588224 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:25:09.588229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:25:09.588232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:25:09.588420 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:25:09.588946 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:09.588953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:09.589274 (+ 321us) client_negotiation.cc:769] Sending connection context
0812 11:25:09.589533 (+ 259us) client_negotiation.cc:240] Negotiation successful
0812 11:25:09.590048 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":273,"threads_started":1}
I20260812 11:25:09.592826 30578 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40469
I20260812 11:25:09.593165 30578 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:09.594094 30578 heartbeater.cc:511] Master 127.12.110.254:40469 requested a full tablet report, sending...
I20260812 11:25:09.596699 30220 ts_manager.cc:194] Registered new tserver with Master: c62c697c53f34b64b2c71a90e1b74d10 (127.12.110.193:39555)
I20260812 11:25:09.598984 30220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:56797
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:25:09.648733 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40469
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:25:09.998090 30586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:09.998354 30586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:09.998459 30586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:10.005721 30586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:10.005824 30586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:10.005942 30586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:25:10.015465 30586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:10.016705 30586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:10.017957 30586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:10.031160 30593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:10.031220 30594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:10.031198 30597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:10.032644 30586 server_base.cc:1034] running on GCE node
I20260812 11:25:10.033396 30586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:10.034431 30586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:10.035725 30586 hybrid_clock.cc:648] HybridClock initialized: now 1786533910035665 us; error 53 us; skew 500 ppm
I20260812 11:25:10.036149 30586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:25:10 dist-test-slave-h6n0 krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:25:10.039392 30586 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:25:10.041281 30586 webserver.cc:460] Webserver started at http://127.12.110.194:34749/ using document root <none> and password file <none>
I20260812 11:25:10.042411 30586 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:10.042645 30586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:10.043125 30586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:10.046361 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "3bee278cba7e458ea20f6a8eabde8a45"
format_stamp: "Formatted at 2026-08-12 11:25:10 on dist-test-slave-h6n0"
server_key: "c67e6a3d1124787edc36c3ba2b81e786"
server_key_iv: "4491092352df1702cf61222d1263499e"
server_key_version: "encryptionkey@0"
I20260812 11:25:10.047351 30586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "3bee278cba7e458ea20f6a8eabde8a45"
format_stamp: "Formatted at 2026-08-12 11:25:10 on dist-test-slave-h6n0"
server_key: "c67e6a3d1124787edc36c3ba2b81e786"
server_key_iv: "4491092352df1702cf61222d1263499e"
server_key_version: "encryptionkey@0"
I20260812 11:25:10.052906 30586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:25:10.056944 30614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:10.058274 30586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:25:10.058502 30586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "3bee278cba7e458ea20f6a8eabde8a45"
format_stamp: "Formatted at 2026-08-12 11:25:10 on dist-test-slave-h6n0"
server_key: "c67e6a3d1124787edc36c3ba2b81e786"
server_key_iv: "4491092352df1702cf61222d1263499e"
server_key_version: "encryptionkey@0"
I20260812 11:25:10.058771 30586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:10.078863 30586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:10.083841 30586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:10.084197 30586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:10.085212 30586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:10.086723 30586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:10.086823 30586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:10.086973 30586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:10.087055 30586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:10.125530 30586 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:34341
I20260812 11:25:10.125566 30823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:34341 every 8 connection(s)
I20260812 11:25:10.126916 30586 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:25:10.130678 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 30586
I20260812 11:25:10.130898 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:25:10.131363 12731 external_mini_cluster.cc:1428] Setting key ec5440173b0e5254f61ce99001abcdac
Aug 12 11:25:10 dist-test-slave-h6n0 krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:25:10.151188 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.129973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:34561 (local address 127.12.110.254:40469)
0812 11:25:10.130522 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:10.130551 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:10.131675 (+ 1124us) server_negotiation.cc:407] Connection header received
0812 11:25:10.133674 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:10.133680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:10.133770 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:10.133896 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:10.135692 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.136946 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.138492 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.139238 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.139389 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.142960 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:10.142969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:10.142977 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:10.143021 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:10.147135 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:10.148148 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:10.148152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:10.148155 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:10.148283 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:10.149027 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:10.149037 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:10.149046 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:10.149503 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:10.149732 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:10.150619 (+ 887us) server_negotiation.cc:299] Negotiation successful
0812 11:25:10.150862 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260812 11:25:10.152595 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.130596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40469 (local address 127.12.110.194:34561)
0812 11:25:10.131491 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:10.131542 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:10.133418 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:10.134090 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:10.134098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:10.134701 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:10.135505 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.135522 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.137106 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.137110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.138340 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.138353 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.139467 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.139473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.139656 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.140359 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:25:10.140386 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:10.142733 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:10.147307 (+ 4574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:25:10.147315 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:25:10.147339 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:25:10.147963 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:25:10.148396 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:25:10.148403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:25:10.148408 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:25:10.148671 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:25:10.149660 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:10.149671 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:10.150120 (+ 449us) client_negotiation.cc:769] Sending connection context
0812 11:25:10.150455 (+ 335us) client_negotiation.cc:240] Negotiation successful
0812 11:25:10.151085 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":402,"threads_started":1}
I20260812 11:25:10.154335 30826 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40469
I20260812 11:25:10.154933 30826 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:10.155983 30826 heartbeater.cc:511] Master 127.12.110.254:40469 requested a full tablet report, sending...
I20260812 11:25:10.157809 30220 ts_manager.cc:194] Registered new tserver with Master: 3bee278cba7e458ea20f6a8eabde8a45 (127.12.110.194:34341)
I20260812 11:25:10.159119 30220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:34561
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:25:10.206321 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40469
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:44239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:25:10.559711 30832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:25:10.559993 30832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:25:10.560154 30832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:25:10.568447 30832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:25:10.568614 30832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:25:10.568809 30832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:25:10.578506 30832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:44239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:25:10.579769 30832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:25:10.581032 30832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:25:10.593784 30842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:10.593961 30845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:25:10.593991 30847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:25:10.595000 30832 server_base.cc:1034] running on GCE node
I20260812 11:25:10.595656 30832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:25:10.596596 30832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:25:10.597870 30832 hybrid_clock.cc:648] HybridClock initialized: now 1786533910597826 us; error 48 us; skew 500 ppm
I20260812 11:25:10.598234 30832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:25:10.603200 30578 heartbeater.cc:503] Master 127.12.110.254:40469 was elected leader, sending a full tablet report...
Aug 12 11:25:10 dist-test-slave-h6n0 krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:25:10.606773 30832 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:25:10.608325 30832 webserver.cc:460] Webserver started at http://127.12.110.195:36145/ using document root <none> and password file <none>
I20260812 11:25:10.609051 30832 fs_manager.cc:359] Metadata directory not provided
I20260812 11:25:10.609217 30832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:25:10.609557 30832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:25:10.612639 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "eda69bf6135745ef90a6264956ce3e10"
format_stamp: "Formatted at 2026-08-12 11:25:10 on dist-test-slave-h6n0"
server_key: "0dbe55a5797f09846615ae9b663309b2"
server_key_iv: "b42c3fc3977fd7b0be4507c908eb1c36"
server_key_version: "encryptionkey@0"
I20260812 11:25:10.613490 30832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "eda69bf6135745ef90a6264956ce3e10"
format_stamp: "Formatted at 2026-08-12 11:25:10 on dist-test-slave-h6n0"
server_key: "0dbe55a5797f09846615ae9b663309b2"
server_key_iv: "b42c3fc3977fd7b0be4507c908eb1c36"
server_key_version: "encryptionkey@0"
I20260812 11:25:10.618917 30832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:25:10.622426 30861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:10.623735 30832 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:25:10.623945 30832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "eda69bf6135745ef90a6264956ce3e10"
format_stamp: "Formatted at 2026-08-12 11:25:10 on dist-test-slave-h6n0"
server_key: "0dbe55a5797f09846615ae9b663309b2"
server_key_iv: "b42c3fc3977fd7b0be4507c908eb1c36"
server_key_version: "encryptionkey@0"
I20260812 11:25:10.624166 30832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:25:10.645744 30832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:25:10.650274 30832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:25:10.650579 30832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:25:10.651584 30832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:25:10.653023 30832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:25:10.653116 30832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:10.653209 30832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:25:10.653261 30832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:25:10.688462 30832 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:44421
I20260812 11:25:10.688572 31069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:44421 every 8 connection(s)
I20260812 11:25:10.689792 30832 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:25:10.692276 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 30832
I20260812 11:25:10.692508 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:25:10.693216 12731 external_mini_cluster.cc:1428] Setting key 27947f8f535523ae4c3f84b14c192398
Aug 12 11:25:10 dist-test-slave-h6n0 krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533910, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:25:10.709897 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.692890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:53695 (local address 127.12.110.254:40469)
0812 11:25:10.693485 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:10.693491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:10.694492 (+ 1001us) server_negotiation.cc:407] Connection header received
0812 11:25:10.696360 (+ 1868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:10.696364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:10.696435 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:10.696533 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:10.697967 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.698821 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.700259 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.700627 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.700718 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.704040 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:10.704045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:10.704050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:10.704112 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:10.706862 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:10.707568 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:10.707575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:10.707584 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:10.707703 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:10.708168 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:10.708174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:10.708178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:10.708448 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:10.708617 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:10.709272 (+ 655us) server_negotiation.cc:299] Negotiation successful
0812 11:25:10.709497 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":325,"threads_started":1}
I20260812 11:25:10.710857 31076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.693477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40469 (local address 127.12.110.195:53695)
0812 11:25:10.694349 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:10.694393 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:10.696131 (+ 1738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:10.696732 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:10.696740 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:10.697281 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:25:10.697820 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.697836 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.698947 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.698951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.700137 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.700149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.700802 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.700806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.701006 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.701630 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:25:10.701651 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:25:10.703769 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:25:10.707014 (+ 3245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:25:10.707023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:25:10.707037 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:25:10.707436 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:25:10.707822 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:25:10.707828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:25:10.707832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:25:10.708032 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:25:10.708591 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:25:10.708599 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:25:10.708908 (+ 309us) client_negotiation.cc:769] Sending connection context
0812 11:25:10.709181 (+ 273us) client_negotiation.cc:240] Negotiation successful
0812 11:25:10.709667 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":349,"threads_started":1}
I20260812 11:25:10.712168 31070 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40469
I20260812 11:25:10.712487 31070 heartbeater.cc:464] Registering TS with master...
I20260812 11:25:10.713263 31070 heartbeater.cc:511] Master 127.12.110.254:40469 requested a full tablet report, sending...
I20260812 11:25:10.714740 30220 ts_manager.cc:194] Registered new tserver with Master: eda69bf6135745ef90a6264956ce3e10 (127.12.110.195:44421)
I20260812 11:25:10.715544 30220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:53695
I20260812 11:25:10.720578 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:25:10.737206 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.725249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.12.110.254:40469)
0812 11:25:10.725411 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:10.725415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:10.725788 (+ 373us) server_negotiation.cc:407] Connection header received
0812 11:25:10.725912 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:10.725916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:10.725979 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:10.726073 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:10.727556 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.728318 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.729640 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.730018 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.730130 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.732411 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:10.732415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:10.732418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:10.732448 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:10.734890 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:10.735516 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:10.735520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:10.735522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:10.735633 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:10.735976 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:10.735979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:10.735981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:10.736291 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:10.736481 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:10.736812 (+ 331us) server_negotiation.cc:299] Negotiation successful
0812 11:25:10.736970 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:25:10.741248 30220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:25:10.750949 30220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:25:10.772656 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.764812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.12.110.194:34341)
0812 11:25:10.766453 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:10.766465 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:10.766495 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:25:10.766635 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:10.766644 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:10.766885 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:10.767164 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:25:10.767735 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.768772 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.770550 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.771472 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.771657 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.771840 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:10.772132 (+ 292us) server_negotiation.cc:299] Negotiation successful
0812 11:25:10.772296 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1484,"thread_start_us":1387,"threads_started":1}
I20260812 11:25:10.772658 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.764683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:34341 (local address 127.0.0.1:40908)
0812 11:25:10.765576 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:10.765624 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:10.765862 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:10.767165 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:10.767172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:10.767199 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:25:10.767582 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.767592 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.768908 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.768916 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.770363 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.770376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.771673 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.771679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.771887 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.771943 (+ 56us) client_negotiation.cc:769] Sending connection context
0812 11:25:10.772079 (+ 136us) client_negotiation.cc:240] Negotiation successful
0812 11:25:10.772278 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":1876,"threads_started":1}
I20260812 11:25:10.777813 30704 tablet_service.cc:1467] Processing CreateTablet for tablet abc57b3316714127a1d18db8ddcdbe5f (DEFAULT_TABLE table=test-table [id=321894a35a8a4ad0bce3f04a9253ead4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:25:10.779317 30704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc57b3316714127a1d18db8ddcdbe5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:25:10.781602 31094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.766913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:39555 (local address 127.0.0.1:35212)
0812 11:25:10.773196 (+ 6283us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:10.773210 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:10.773294 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:10.774698 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:10.774705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:10.774731 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:25:10.774915 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.774931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.777082 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.777088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.778941 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.778956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.780872 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.780879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.781050 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.781084 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:25:10.781161 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 11:25:10.781318 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6215,"thread_start_us":6549,"threads_started":1}
I20260812 11:25:10.782896 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.767184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.12.110.193:39555)
0812 11:25:10.774062 (+ 6878us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:10.774073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:10.774103 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:25:10.774200 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:10.774207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:10.774437 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:10.775107 (+ 670us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:25:10.775370 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.776889 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.779201 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.780672 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.782136 (+ 1464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.782308 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:10.782374 (+ 66us) server_negotiation.cc:299] Negotiation successful
0812 11:25:10.782584 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6694,"thread_start_us":6514,"threads_started":1}
I20260812 11:25:10.783185 31092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.764735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:44421 (local address 127.0.0.1:44606)
0812 11:25:10.766811 (+ 2076us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:10.766829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:10.766946 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:10.776795 (+ 9849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:10.776800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:10.776825 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:25:10.777202 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.777213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.779351 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.779357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.780666 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:10.780677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.782453 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.782459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:10.782667 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.782692 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:25:10.782745 (+ 53us) client_negotiation.cc:240] Negotiation successful
0812 11:25:10.782839 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1948,"thread_start_us":396,"threads_started":1}
I20260812 11:25:10.784178 31097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:10.765339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44606 (local address 127.12.110.195:44421)
0812 11:25:10.776359 (+ 11020us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:10.776365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:10.776392 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:25:10.776463 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:10.776467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:10.776613 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:10.776804 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:25:10.777393 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.779229 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.780819 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:10.781963 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:10.783503 (+ 1540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:10.783639 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:10.783688 (+ 49us) server_negotiation.cc:299] Negotiation successful
0812 11:25:10.783821 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10876,"thread_start_us":10710,"threads_started":1}
I20260812 11:25:10.786501 30449 tablet_service.cc:1467] Processing CreateTablet for tablet abc57b3316714127a1d18db8ddcdbe5f (DEFAULT_TABLE table=test-table [id=321894a35a8a4ad0bce3f04a9253ead4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:25:10.787765 30950 tablet_service.cc:1467] Processing CreateTablet for tablet abc57b3316714127a1d18db8ddcdbe5f (DEFAULT_TABLE table=test-table [id=321894a35a8a4ad0bce3f04a9253ead4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:25:10.788847 30449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc57b3316714127a1d18db8ddcdbe5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:25:10.789115 30950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc57b3316714127a1d18db8ddcdbe5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:25:10.793329 31104 tablet_bootstrap.cc:492] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Bootstrap starting.
I20260812 11:25:10.796403 31104 tablet_bootstrap.cc:654] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Neither blocks nor log segments found. Creating new log.
I20260812 11:25:10.797770 31104 log.cc:824] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Log is configured to *not* fsync() on all Append() calls
I20260812 11:25:10.801075 31104 tablet_bootstrap.cc:492] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: No bootstrap required, opened a new log
I20260812 11:25:10.801582 31104 ts_tablet_manager.cc:1397] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260812 11:25:10.803442 31110 tablet_bootstrap.cc:492] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Bootstrap starting.
I20260812 11:25:10.804631 31107 tablet_bootstrap.cc:492] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Bootstrap starting.
I20260812 11:25:10.808262 31107 tablet_bootstrap.cc:654] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Neither blocks nor log segments found. Creating new log.
I20260812 11:25:10.809064 31110 tablet_bootstrap.cc:654] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Neither blocks nor log segments found. Creating new log.
I20260812 11:25:10.809497 31107 log.cc:824] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Log is configured to *not* fsync() on all Append() calls
I20260812 11:25:10.810067 31110 log.cc:824] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Log is configured to *not* fsync() on all Append() calls
I20260812 11:25:10.812732 31107 tablet_bootstrap.cc:492] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: No bootstrap required, opened a new log
I20260812 11:25:10.812983 31110 tablet_bootstrap.cc:492] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: No bootstrap required, opened a new log
I20260812 11:25:10.813030 31107 ts_tablet_manager.cc:1397] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260812 11:25:10.812789 31104 raft_consensus.cc:348] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:10.813246 31104 raft_consensus.cc:374] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:25:10.813369 31104 raft_consensus.cc:729] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bee278cba7e458ea20f6a8eabde8a45, State: Initialized, Role: FOLLOWER
I20260812 11:25:10.813570 31110 ts_tablet_manager.cc:1397] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260812 11:25:10.814038 31104 consensus_queue.cc:260] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:10.815142 30826 heartbeater.cc:503] Master 127.12.110.254:40469 was elected leader, sending a full tablet report...
I20260812 11:25:10.815789 31104 ts_tablet_manager.cc:1428] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260812 11:25:10.823683 31107 raft_consensus.cc:348] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:10.824147 31107 raft_consensus.cc:374] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:25:10.823967 31110 raft_consensus.cc:348] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:10.824254 31107 raft_consensus.cc:729] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62c697c53f34b64b2c71a90e1b74d10, State: Initialized, Role: FOLLOWER
I20260812 11:25:10.824368 31110 raft_consensus.cc:374] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:25:10.824460 31110 raft_consensus.cc:729] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda69bf6135745ef90a6264956ce3e10, State: Initialized, Role: FOLLOWER
I20260812 11:25:10.824889 31107 consensus_queue.cc:260] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:10.824976 31110 consensus_queue.cc:260] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:10.825978 31070 heartbeater.cc:503] Master 127.12.110.254:40469 was elected leader, sending a full tablet report...
I20260812 11:25:10.826784 31110 ts_tablet_manager.cc:1428] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260812 11:25:10.826980 31107 ts_tablet_manager.cc:1428] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260812 11:25:10.879065 30827 tablet.cc:2369] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:25:10.941509 31071 tablet.cc:2369] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:25:11.045809 31114 raft_consensus.cc:482] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:25:11.046208 31114 raft_consensus.cc:504] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:11.047982 31114 leader_election.cc:288] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eda69bf6135745ef90a6264956ce3e10 (127.12.110.195:44421), c62c697c53f34b64b2c71a90e1b74d10 (127.12.110.193:39555)
I20260812 11:25:11.055781 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.048585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:52957 (local address 127.12.110.193:39555)
0812 11:25:11.048835 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:11.048844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:11.049717 (+ 873us) server_negotiation.cc:407] Connection header received
0812 11:25:11.049912 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:11.049917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:11.049990 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:11.050093 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:25:11.050851 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.051534 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.053777 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.054770 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.054894 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.054949 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:11.055381 (+ 432us) server_negotiation.cc:299] Negotiation successful
0812 11:25:11.055530 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:25:11.055855 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.048413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:39555 (local address 127.12.110.194:52957)
0812 11:25:11.049585 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:11.049603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:11.049753 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:11.050165 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:11.050168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:11.050199 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:25:11.050693 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:11.050706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.051711 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.051715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:11.053436 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:11.053455 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.055000 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.055008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:11.055190 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.055232 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:25:11.055335 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 11:25:11.055439 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1034,"mutex_wait_us":123,"thread_start_us":643,"threads_started":1}
I20260812 11:25:11.056095 31097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.048500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:50563 (local address 127.12.110.195:44421)
0812 11:25:11.048780 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:11.048790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:11.049545 (+ 755us) server_negotiation.cc:407] Connection header received
0812 11:25:11.049716 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:11.049722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:11.049810 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:11.049928 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:25:11.050715 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.051520 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.054086 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.055061 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.055219 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.055288 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:11.055681 (+ 393us) server_negotiation.cc:299] Negotiation successful
0812 11:25:11.055824 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:25:11.056216 31118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.048282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:44421 (local address 127.12.110.194:50563)
0812 11:25:11.049428 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0812 11:25:11.049444 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:25:11.049569 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:25:11.049944 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:25:11.049948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:25:11.049971 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:25:11.050497 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:11.050511 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.051662 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.051669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:11.053868 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:25:11.053883 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.055278 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.055291 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:25:11.055501 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.055533 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:25:11.055633 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 11:25:11.055767 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":148,"spinlock_wait_cycles":21888,"thread_start_us":825,"threads_started":1}
I20260812 11:25:11.057087 30481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc57b3316714127a1d18db8ddcdbe5f" candidate_uuid: "3bee278cba7e458ea20f6a8eabde8a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62c697c53f34b64b2c71a90e1b74d10" is_pre_election: true
I20260812 11:25:11.057200 30985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc57b3316714127a1d18db8ddcdbe5f" candidate_uuid: "3bee278cba7e458ea20f6a8eabde8a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda69bf6135745ef90a6264956ce3e10" is_pre_election: true
I20260812 11:25:11.057649 30481 raft_consensus.cc:2454] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bee278cba7e458ea20f6a8eabde8a45 in term 0.
I20260812 11:25:11.057705 30985 raft_consensus.cc:2454] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bee278cba7e458ea20f6a8eabde8a45 in term 0.
I20260812 11:25:11.058410 30616 leader_election.cc:302] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bee278cba7e458ea20f6a8eabde8a45, c62c697c53f34b64b2c71a90e1b74d10, eda69bf6135745ef90a6264956ce3e10; no voters:
I20260812 11:25:11.058925 31114 raft_consensus.cc:2781] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:25:11.059077 31114 raft_consensus.cc:482] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:25:11.059201 31114 raft_consensus.cc:3037] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:25:11.061371 31114 raft_consensus.cc:504] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
I20260812 11:25:11.062090 31114 leader_election.cc:288] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [CANDIDATE]: Term 1 election: Requested vote from peers eda69bf6135745ef90a6264956ce3e10 (127.12.110.195:44421), c62c697c53f34b64b2c71a90e1b74d10 (127.12.110.193:39555)
I20260812 11:25:11.062932 30985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc57b3316714127a1d18db8ddcdbe5f" candidate_uuid: "3bee278cba7e458ea20f6a8eabde8a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda69bf6135745ef90a6264956ce3e10"
I20260812 11:25:11.063088 30481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc57b3316714127a1d18db8ddcdbe5f" candidate_uuid: "3bee278cba7e458ea20f6a8eabde8a45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62c697c53f34b64b2c71a90e1b74d10"
I20260812 11:25:11.063287 30985 raft_consensus.cc:3037] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:25:11.063356 30481 raft_consensus.cc:3037] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:25:11.067205 30481 raft_consensus.cc:2454] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bee278cba7e458ea20f6a8eabde8a45 in term 1.
I20260812 11:25:11.067211 30985 raft_consensus.cc:2454] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bee278cba7e458ea20f6a8eabde8a45 in term 1.
I20260812 11:25:11.067749 30616 leader_election.cc:302] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bee278cba7e458ea20f6a8eabde8a45, c62c697c53f34b64b2c71a90e1b74d10; no voters:
I20260812 11:25:11.068089 31114 raft_consensus.cc:2781] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:25:11.068619 31114 raft_consensus.cc:686] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 1 LEADER]: Becoming Leader. State: Replica: 3bee278cba7e458ea20f6a8eabde8a45, State: Running, Role: LEADER
I20260812 11:25:11.069192 31114 consensus_queue.cc:237] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } }
W20260812 11:25:11.072227 30579 tablet.cc:2369] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:25:11.075392 30220 catalog_manager.cc:5495] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bee278cba7e458ea20f6a8eabde8a45 (127.12.110.194). New cstate: current_term: 1 leader_uuid: "3bee278cba7e458ea20f6a8eabde8a45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bee278cba7e458ea20f6a8eabde8a45" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 34341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 } health_report { overall_health: UNKNOWN } } }
I20260812 11:25:11.152391 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.146113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.12.110.194:34341)
0812 11:25:11.146289 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:11.146294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:11.146314 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:25:11.146442 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:11.146446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:11.146518 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:11.146720 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:25:11.147118 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.147841 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.149893 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.150486 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.150668 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.151267 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:25:11.151751 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:25:11.151868 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:11.152100 (+ 232us) server_negotiation.cc:299] Negotiation successful
0812 11:25:11.152167 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260812 11:25:11.160619 30481 raft_consensus.cc:1261] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 1 FOLLOWER]: Refusing update from remote peer 3bee278cba7e458ea20f6a8eabde8a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:25:11.160620 30985 raft_consensus.cc:1261] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 1 FOLLOWER]: Refusing update from remote peer 3bee278cba7e458ea20f6a8eabde8a45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:25:11.161753 31120 consensus_queue.cc:1035] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c62c697c53f34b64b2c71a90e1b74d10" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:25:11.162129 31114 consensus_queue.cc:1035] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eda69bf6135745ef90a6264956ce3e10" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:25:11.174226 31130 mvcc.cc:205] Tried to move back new op lower bound from 7317642900103655424 to 7317642899743477760. Current Snapshot: MvccSnapshot[applied={T|T < 7317642900103655424}]
I20260812 11:25:11.193048 30220 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:25:11.193287 30220 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:25:11.197009 30220 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edd4dca6da4e49edbcf78a3453a9d401: Sending DeleteTablet for 3 replicas of tablet abc57b3316714127a1d18db8ddcdbe5f
I20260812 11:25:11.198334 30950 tablet_service.cc:1514] Processing DeleteTablet for tablet abc57b3316714127a1d18db8ddcdbe5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:25:11 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:44606
I20260812 11:25:11.198351 30704 tablet_service.cc:1514] Processing DeleteTablet for tablet abc57b3316714127a1d18db8ddcdbe5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:25:11 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:40908
I20260812 11:25:11.199072 30449 tablet_service.cc:1514] Processing DeleteTablet for tablet abc57b3316714127a1d18db8ddcdbe5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:25:11 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:35212
I20260812 11:25:11.202497 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 30335
I20260812 11:25:11.213667 31145 tablet_replica.cc:330] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: stopping tablet replica
I20260812 11:25:11.214313 31145 raft_consensus.cc:2229] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:25:11.215013 31145 raft_consensus.cc:2258] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:25:11.218741 31145 ts_tablet_manager.cc:1905] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:25:11.219645 31142 tablet_replica.cc:330] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: stopping tablet replica
I20260812 11:25:11.220184 31142 raft_consensus.cc:2229] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:25:11.221804 31142 raft_consensus.cc:2258] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:25:11.226054 31145 ts_tablet_manager.cc:1918] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:25:11.226220 31145 log.cc:1196] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/abc57b3316714127a1d18db8ddcdbe5f
I20260812 11:25:11.226809 31145 ts_tablet_manager.cc:1939] T abc57b3316714127a1d18db8ddcdbe5f P c62c697c53f34b64b2c71a90e1b74d10: Deleting consensus metadata
I20260812 11:25:11.228482 31142 ts_tablet_manager.cc:1905] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:25:11.229100 30190 catalog_manager.cc:4841] TS c62c697c53f34b64b2c71a90e1b74d10 (127.12.110.193:39555): tablet abc57b3316714127a1d18db8ddcdbe5f (table test-table [id=321894a35a8a4ad0bce3f04a9253ead4]) successfully deleted
I20260812 11:25:11.230947 31144 tablet_replica.cc:330] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: stopping tablet replica
I20260812 11:25:11.231391 31144 raft_consensus.cc:2229] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:25:11.232160 31144 raft_consensus.cc:2258] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:25:11.234481 31144 ts_tablet_manager.cc:1905] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 11:25:11 dist-test-slave-h6n0 krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:25:11.238256 31142 ts_tablet_manager.cc:1918] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:25:11.238453 31142 log.cc:1196] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/abc57b3316714127a1d18db8ddcdbe5f
I20260812 11:25:11.239058 31142 ts_tablet_manager.cc:1939] T abc57b3316714127a1d18db8ddcdbe5f P 3bee278cba7e458ea20f6a8eabde8a45: Deleting consensus metadata
I20260812 11:25:11.240826 30190 catalog_manager.cc:4841] TS 3bee278cba7e458ea20f6a8eabde8a45 (127.12.110.194:34341): tablet abc57b3316714127a1d18db8ddcdbe5f (table test-table [id=321894a35a8a4ad0bce3f04a9253ead4]) successfully deleted
I20260812 11:25:11.241361 31144 ts_tablet_manager.cc:1918] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:25:11.241472 31144 log.cc:1196] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/abc57b3316714127a1d18db8ddcdbe5f
I20260812 11:25:11.241890 31144 ts_tablet_manager.cc:1939] T abc57b3316714127a1d18db8ddcdbe5f P eda69bf6135745ef90a6264956ce3e10: Deleting consensus metadata
I20260812 11:25:11.243918 30191 catalog_manager.cc:4841] TS eda69bf6135745ef90a6264956ce3e10 (127.12.110.195:44421): tablet abc57b3316714127a1d18db8ddcdbe5f (table test-table [id=321894a35a8a4ad0bce3f04a9253ead4]) successfully deleted
I20260812 11:25:11.245527 31103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.203827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35214 (local address 127.12.110.193:39555)
0812 11:25:11.204051 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:11.204060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:11.216491 (+ 12431us) server_negotiation.cc:407] Connection header received
0812 11:25:11.217508 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:11.217517 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:11.217616 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:11.217696 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:11.222027 (+ 4331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.223096 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.225263 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.225954 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.226083 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.237970 (+ 11887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:11.237990 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:11.238009 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:11.238059 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:11.242123 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:11.243140 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:11.243151 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:11.243155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:11.243335 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:11.243806 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:11.243810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:11.243812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:11.244250 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:11.244558 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:11.244902 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 11:25:11.245143 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:25:11.246196 30570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:11.339184 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 30335
I20260812 11:25:11.371587 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 30586
Aug 12 11:25:11 dist-test-slave-h6n0 krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:25:11.388849 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.372461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.12.110.194:34341)
0812 11:25:11.372640 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:11.372644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:11.372662 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:25:11.372870 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:11.372873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:11.372938 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:11.373021 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:11.374695 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.375501 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.377308 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.377729 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.377820 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.382938 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:11.382950 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:11.382965 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:11.383001 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:11.386006 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:11.386770 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:11.386784 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:11.386787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:11.386934 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:11.387314 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:11.387318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:11.387320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:11.387649 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:11.388022 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:11.388253 (+ 231us) server_negotiation.cc:299] Negotiation successful
0812 11:25:11.388477 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:25:11.389627 30811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:11.502045 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 30586
I20260812 11:25:11.528613 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 30832
Aug 12 11:25:11 dist-test-slave-h6n0 krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:25:11.544255 31097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:25:11.529425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44618 (local address 127.12.110.195:44421)
0812 11:25:11.529623 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:25:11.529631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:25:11.529727 (+ 96us) server_negotiation.cc:407] Connection header received
0812 11:25:11.529950 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:25:11.529958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:25:11.530036 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:25:11.530165 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:25:11.531548 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.532277 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.533566 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:25:11.533958 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:25:11.534100 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:25:11.538666 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:25:11.538679 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:25:11.538694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:25:11.538726 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:25:11.541446 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:11.542149 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:11.542157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:11.542159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:11.542293 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:25:11.542723 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:25:11.542728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:25:11.542733 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:25:11.543082 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:25:11.543410 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0812 11:25:11.543713 (+ 303us) server_negotiation.cc:299] Negotiation successful
0812 11:25:11.543932 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:25:11.544986 31060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:11.636982 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 30832
I20260812 11:25:11.669026 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 30163
I20260812 11:25:11.670217 30276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:25:11.784152 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 30163
2026-08-12T11:25:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5074 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:25:11 dist-test-slave-h6n0 krb5kdc[31163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:25:11 dist-test-slave-h6n0 krb5kdc[31163](info): set up 2 sockets
Aug 12 11:25:11 dist-test-slave-h6n0 krb5kdc[31163](info): commencing operation
krb5kdc: starting...
W20260812 11:25:13.200908 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.361s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:25:13 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:25:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:25:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/rangeradmin_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/rangeradmin_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/rangerlookup_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/rangerlookup_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
I20260812 11:25:13.653846 12731 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklUqj9l/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/postgres -l logfile start
2026-08-12 11:25:16.315 UTC [31222] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 11:25:16.315 UTC [31222] LOG: listening on IPv4 address "127.12.110.212", port 57183
2026-08-12 11:25:16.318 UTC [31222] LOG: listening on Unix socket "/tmp/.s.PGSQL.57183"
2026-08-12 11:25:16.344 UTC [31225] LOG: database system was shut down at 2026-08-12 11:25:14 UTC
2026-08-12 11:25:16.348 UTC [31222] LOG: database system is ready to accept connections
I20260812 11:25:17.671100 12731 mini_postgres.cc:96] Postgres bound to 57183
2026-08-12 11:25:17.699 UTC [31246] FATAL: database "slave" does not exist
127.12.110.212:57183 - accepting connections
I20260812 11:25:17.700057 12731 mini_ranger.cc:161] Starting Ranger...
I20260812 11:25:17.732146 12731 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 11:25:18.250778 12731 mini_ranger.cc:87] Created ranger Postgres database
I20260812 11:25:18.250962 12731 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-admin
2026-08-12 11:25:18,620 [I] DB FLAVOR :POSTGRES
2026-08-12 11:25:18,621 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:25:18,621 [I] Checking connection..
2026-08-12 11:25:18,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 11:25:18,878 [I] Checking connection passed.
2026-08-12 11:25:18,878 [I] --------- Verifying version history table ---------
2026-08-12 11:25:18,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:25:19,129 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:25:19,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:25:19,377 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:25:19,378 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 11:25:19,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 11:25:19.620 UTC [31362] WARNING: there is no transaction in progress
2026-08-12 11:25:19,629 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 11:25:19,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:25:19,902 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 11:25:19,902 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:25:19,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 11:25:20,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 11:25:20,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h6n0.c.gcp-upstream.internal','N') ;"
2026-08-12 11:25:20,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:25:20,917 [I] Table x_portal_user does not exist in database ranger
2026-08-12 11:25:20,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:25:21,174 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 11:25:21,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:25:21,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:25:21,697 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 11:25:21,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 11:25:22.678 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.695 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.711 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.726 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.741 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.761 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.817 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.824 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.838 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.848 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.858 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.870 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.879 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.886 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.893 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:22.901 UTC [31549] WARNING: there is no transaction in progress
2026-08-12 11:25:23,565 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 11:25:23,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 11:25:23,801 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 11:25:23,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:25:24,062 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 11:25:24,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:25:24,337 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 11:25:24,338 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:25:24,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:25:24,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h6n0.c.gcp-upstream.internal';"
2026-08-12 11:25:25,108 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 11:25:25,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 11:25:25,380 [I] DB_PATCHES have already been applied
I20260812 11:25:25.387233 12731 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklUqj9l/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 11:25:25.387331 12731 mini_ranger.cc:193] Using host: 127.12.110.212
I20260812 11:25:25.413416 12731 mini_ranger.cc:227] Ranger admin URL: http://127.12.110.212:54025
Aug 12, 2026 11:25:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:25:25 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:25:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 11:25:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 11:25:25 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 11:25:26 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 11:25:26 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 11:25:26.913976 12731 mini_ranger.cc:160] Time spent starting Ranger: real 9.214s user 0.000s sys 0.074s
Aug 12, 2026 11:25:48 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 11:25:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Aug 12, 2026 11:25:48 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 11:25:48 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 11:25:49 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 11:25:50.228837 12731 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/rangerkms_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/rangerkms_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.110.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.110.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 11:25:50 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533950, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 11:25:50.455760 12731 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 11:25:50.487234 12731 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 11:25:51.424352 12731 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 11:25:51.424644 12731 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms
I20260812 11:25:51.424913 12731 mini_ranger_kms.cc:227] Using postgres at 127.12.110.212:57183
2026-08-12 11:25:51,951 [I] DB FLAVOR :POSTGRES
2026-08-12 11:25:51,951 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:25:51,951 [I] Checking connection
2026-08-12 11:25:51,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 11:25:52,199 [I] connection success
2026-08-12 11:25:52,199 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 11:25:52,199 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 11:25:52,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 11:25:52,485 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 11:25:52,486 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:25:52,486 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 11:25:52,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:57183/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 11:25:52,793 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 11:25:53.317408 12731 mini_ranger_kms.cc:310] Created kms service
I20260812 11:25:53.526156 12731 mini_ranger_kms.cc:326] Created kudu user
I20260812 11:25:53.577062 12731 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 11:25:53 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533950, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:25:53.997254 12731 mini_ranger_kms.cc:384] Added ranger policy
I20260812 11:25:53.998724 12731 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 11:25:53.998816 12731 mini_ranger_kms.cc:240] Using host: 127.12.110.212
I20260812 11:25:54.035326 12731 mini_ranger_kms.cc:276] Ranger KMS PID: 31904
I20260812 11:25:54.035771 12731 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.110.212:41403
Aug 12, 2026 11:25:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:25:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:25:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 11:25:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 11:25:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 11:25:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 11:25:56 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.110.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 11:25:56 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.110.212@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:25:56.969832 12731 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.514s user 0.147s sys 0.436s
I20260812 11:25:56.970203 12731 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 11:25:56.970415 12731 mini_ranger_kms.cc:401] 127.12.110.212:41403/kms/v1/keys
Aug 12 11:25:57 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533956, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.110.212@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
Aug 12 11:25:59 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:25:59.758786 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40559
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:40559
--ranger_config_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:41403/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:26:00.117617 31990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:00.117883 31990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:00.118034 31990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:00.124962 31990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:26:00.125061 31990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:00.125156 31990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:00.125243 31990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:26:00.125316 31990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:26:00.135125 31990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.110.212:41403/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:40559
--ranger_config_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:00.136511 31990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:00.137913 31990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:00.150801 32003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:00.150770 32006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:00.150835 32002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:00.151543 31990 server_base.cc:1034] running on GCE node
I20260812 11:26:00.152318 31990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:00.153934 31990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:00.155238 31990 hybrid_clock.cc:648] HybridClock initialized: now 1786533960155193 us; error 26 us; skew 500 ppm
I20260812 11:26:00.155664 31990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:00 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533960, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:00.161803 31990 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:26:00.164291 31990 webserver.cc:460] Webserver started at http://127.12.110.254:38057/ using document root <none> and password file <none>
I20260812 11:26:00.164917 31990 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:00.165019 31990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:00.165275 31990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:00 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533960, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:00.222368 31990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "45593998800b47d39fa09f268f8aa675"
format_stamp: "Formatted at 2026-08-12 11:26:00 on dist-test-slave-h6n0"
server_key: "b9fa44a3bada1d7fd366a392a57898e3"
server_key_iv: "54ec3a8ea0c6b9c01393dbf4e1e3b8e2"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:00.223397 31990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "45593998800b47d39fa09f268f8aa675"
format_stamp: "Formatted at 2026-08-12 11:26:00 on dist-test-slave-h6n0"
server_key: "b9fa44a3bada1d7fd366a392a57898e3"
server_key_iv: "54ec3a8ea0c6b9c01393dbf4e1e3b8e2"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:00.229002 31990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:26:00.250789 32019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:00.252687 31990 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:26:00.252890 31990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "45593998800b47d39fa09f268f8aa675"
format_stamp: "Formatted at 2026-08-12 11:26:00 on dist-test-slave-h6n0"
server_key: "b9fa44a3bada1d7fd366a392a57898e3"
server_key_iv: "54ec3a8ea0c6b9c01393dbf4e1e3b8e2"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:00.253226 31990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:00.296941 31990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:00.301635 31990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:00.301963 31990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:00.325145 31990 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:40559
I20260812 11:26:00.325196 32102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:40559 every 8 connection(s)
I20260812 11:26:00.326682 31990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:26:00.329957 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 31990
I20260812 11:26:00.330201 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:26:00.330716 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:26:00.331039 12731 external_mini_cluster.cc:1428] Setting key 93d06e8990f03755f94c89b88f52b2c9
I20260812 11:26:00.345886 32103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675: Bootstrap starting.
Aug 12 11:26:00 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:00.349246 32103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675: Neither blocks nor log segments found. Creating new log.
I20260812 11:26:00.350320 32103 log.cc:824] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675: Log is configured to *not* fsync() on all Append() calls
I20260812 11:26:00.353308 32103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675: No bootstrap required, opened a new log
I20260812 11:26:00.355392 32107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:00.333079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.12.110.254:40559)
0812 11:26:00.333914 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:00.333926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:00.333970 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:26:00.335850 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:00.335870 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:00.336245 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:00.336556 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:00.338281 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:00.340411 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:00.342286 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:00.343114 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:00.343253 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:00.348098 (+ 4845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:00.348107 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:00.348121 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:00.348164 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:00.351443 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:00.352137 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:00.352144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:00.352149 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:00.352299 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:00.352694 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:00.352699 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:00.352702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:00.353219 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:00.353526 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:00.354012 (+ 486us) server_negotiation.cc:299] Negotiation successful
0812 11:26:00.354388 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":411,"threads_started":1}
I20260812 11:26:00.361862 32103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45593998800b47d39fa09f268f8aa675" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40559 } }
I20260812 11:26:00.362279 32103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:26:00.362408 32103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45593998800b47d39fa09f268f8aa675, State: Initialized, Role: FOLLOWER
I20260812 11:26:00.363106 32103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45593998800b47d39fa09f268f8aa675" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40559 } }
I20260812 11:26:00.363396 32103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:26:00.363543 32103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:26:00.363785 32103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:26:00.366933 32103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45593998800b47d39fa09f268f8aa675" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40559 } }
I20260812 11:26:00.367497 32103 leader_election.cc:302] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45593998800b47d39fa09f268f8aa675; no voters:
I20260812 11:26:00.368029 32103 leader_election.cc:288] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:26:00.368212 32115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:26:00.369828 32115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [term 1 LEADER]: Becoming Leader. State: Replica: 45593998800b47d39fa09f268f8aa675, State: Running, Role: LEADER
I20260812 11:26:00.370345 32115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45593998800b47d39fa09f268f8aa675" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40559 } }
I20260812 11:26:00.370818 32103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:26:00.373292 32117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45593998800b47d39fa09f268f8aa675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45593998800b47d39fa09f268f8aa675" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40559 } } }
I20260812 11:26:00.373238 32118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45593998800b47d39fa09f268f8aa675. Latest consensus state: current_term: 1 leader_uuid: "45593998800b47d39fa09f268f8aa675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45593998800b47d39fa09f268f8aa675" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40559 } } }
I20260812 11:26:00.373718 32117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [sys.catalog]: This master's current role is: LEADER
I20260812 11:26:00.373759 32118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675 [sys.catalog]: This master's current role is: LEADER
I20260812 11:26:00.375535 32134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:26:00.380816 32134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:26:00.381556 32103 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:26:00.400460 32134 catalog_manager.cc:1347] Generated new cluster ID: 89f176347a7145f886ca2c0ed5ca7355
I20260812 11:26:00.400669 32134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:26:00.436158 32134 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:26:00.437624 32134 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:26:00.444708 32134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675: Generated new TSK 0
I20260812 11:26:00.445596 32134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:26:01.284003 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40559
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:26:01 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786533961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 11:26:01.697055 32193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:01.697311 32193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:01.697443 32193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:01.705199 32193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:01.705304 32193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:01.705516 32193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:26:01.717657 32193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.110.212:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:01.719485 32193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:01.720889 32193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:01.737460 32208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:01.744625 32209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:01.745585 32193 server_base.cc:1034] running on GCE node
W20260812 11:26:01.746070 32212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:01.746978 32193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:01.751137 32193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:01.752461 32193 hybrid_clock.cc:648] HybridClock initialized: now 1786533961752385 us; error 61 us; skew 500 ppm
I20260812 11:26:01.752871 32193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:01 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:01.757406 32193 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:26:01.763458 32193 webserver.cc:460] Webserver started at http://127.12.110.193:33371/ using document root <none> and password file <none>
I20260812 11:26:01.764504 32193 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:01.764689 32193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:01.765117 32193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:01 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:01.785791 32193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "778764dad1fc4ba89318ee8df1ccef3e"
format_stamp: "Formatted at 2026-08-12 11:26:01 on dist-test-slave-h6n0"
server_key: "e266161d5e3e052792ddad141b465d2f"
server_key_iv: "1920a5ae6b75b9cff065ca2bddcbd498"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:01.789086 32193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "778764dad1fc4ba89318ee8df1ccef3e"
format_stamp: "Formatted at 2026-08-12 11:26:01 on dist-test-slave-h6n0"
server_key: "e266161d5e3e052792ddad141b465d2f"
server_key_iv: "1920a5ae6b75b9cff065ca2bddcbd498"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:01.803638 32193 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.009s sys 0.004s
I20260812 11:26:01.843235 32225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:01.845346 32193 fs_manager.cc:725] Time spent opening block manager: real 0.022s user 0.008s sys 0.011s
I20260812 11:26:01.845640 32193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "778764dad1fc4ba89318ee8df1ccef3e"
format_stamp: "Formatted at 2026-08-12 11:26:01 on dist-test-slave-h6n0"
server_key: "e266161d5e3e052792ddad141b465d2f"
server_key_iv: "1920a5ae6b75b9cff065ca2bddcbd498"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:01.846040 32193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:01.899595 32193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:01.904104 32193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:01.904445 32193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:01.905503 32193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:26:01.907042 32193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:26:01.907158 32193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:01.907359 32193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:26:01.907441 32193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:01.998879 32193 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:39039
I20260812 11:26:01.998929 32429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:39039 every 8 connection(s)
I20260812 11:26:02.000793 32193 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:26:02.008155 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 32193
I20260812 11:26:02.008373 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:26:02.009182 12731 external_mini_cluster.cc:1428] Setting key c84c3c3774142f0db8f7873e316c7705
Aug 12 11:26:02 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:02.041108 32439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:02.005089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40559 (local address 127.12.110.193:37917)
0812 11:26:02.009653 (+ 4564us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:02.009711 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:02.012147 (+ 2436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:02.015532 (+ 3385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:02.015544 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:02.016214 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:26:02.017470 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:02.017500 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:02.020421 (+ 2921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:02.020428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:02.024563 (+ 4135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:02.024581 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:02.027578 (+ 2997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:02.027586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:02.027904 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:02.028700 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:26:02.028731 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:26:02.031742 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:26:02.035850 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:02.035861 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:02.035879 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:02.036543 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:02.037018 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:02.037022 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:02.037025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:02.037272 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:02.038068 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:26:02.038078 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:26:02.038688 (+ 610us) client_negotiation.cc:769] Sending connection context
0812 11:26:02.039015 (+ 327us) client_negotiation.cc:240] Negotiation successful
0812 11:26:02.039620 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4276,"thread_start_us":4041,"threads_started":1}
I20260812 11:26:02.042347 32435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:02.004278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:37917 (local address 127.12.110.254:40559)
0812 11:26:02.014904 (+ 10626us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:02.014913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:02.014935 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:26:02.014997 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:02.015005 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:02.015089 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:02.015210 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:02.018687 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:02.020008 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:02.025637 (+ 5629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:02.027301 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:02.027455 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:02.031993 (+ 4538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:02.031998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:02.032001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:02.032043 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:02.035667 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:02.036698 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:02.036704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:02.036708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:02.036875 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:02.037431 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:02.037436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:02.037440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:02.037807 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:02.037969 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:02.041698 (+ 3729us) server_negotiation.cc:299] Negotiation successful
0812 11:26:02.041944 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10486,"thread_start_us":10352,"threads_started":1}
I20260812 11:26:02.044780 32431 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40559
I20260812 11:26:02.045123 32431 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:02.046018 32431 heartbeater.cc:511] Master 127.12.110.254:40559 requested a full tablet report, sending...
I20260812 11:26:02.048719 32042 ts_manager.cc:194] Registered new tserver with Master: 778764dad1fc4ba89318ee8df1ccef3e (127.12.110.193:39039)
I20260812 11:26:02.050962 32042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:37917
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:26:02.089960 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40559
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:26:02 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
Aug 12 11:26:02 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533961, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
W20260812 11:26:02.595772 32442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:02.596251 32442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:02.596421 32442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:02.606312 32442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:02.606523 32442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:02.606761 32442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:26:02.620319 32442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.110.212:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:02.622279 32442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:02.624095 32442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:02.663837 32451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:02.664063 32452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:02.666643 32442 server_base.cc:1034] running on GCE node
W20260812 11:26:02.666834 32455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:02.670234 32442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:02.676463 32442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:02.677825 32442 hybrid_clock.cc:648] HybridClock initialized: now 1786533962677782 us; error 37 us; skew 500 ppm
I20260812 11:26:02.678273 32442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:02 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533962, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:02.682699 32442 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:26:02.700475 32442 webserver.cc:460] Webserver started at http://127.12.110.194:39707/ using document root <none> and password file <none>
I20260812 11:26:02.701475 32442 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:02.701714 32442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:02.702145 32442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:02 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533962, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:02.723379 32442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "c95a2e242ffc46d89c2989081ada6fac"
format_stamp: "Formatted at 2026-08-12 11:26:02 on dist-test-slave-h6n0"
server_key: "117ba1e7ab06f1c351145232df7e0092"
server_key_iv: "78343edbe4030c09bc3013f5fd236e12"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:02.724519 32442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "c95a2e242ffc46d89c2989081ada6fac"
format_stamp: "Formatted at 2026-08-12 11:26:02 on dist-test-slave-h6n0"
server_key: "117ba1e7ab06f1c351145232df7e0092"
server_key_iv: "78343edbe4030c09bc3013f5fd236e12"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:02.740419 32442 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.007s sys 0.005s
I20260812 11:26:02.783558 32473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:02.786415 32442 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.000s sys 0.006s
I20260812 11:26:02.786966 32442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "c95a2e242ffc46d89c2989081ada6fac"
format_stamp: "Formatted at 2026-08-12 11:26:02 on dist-test-slave-h6n0"
server_key: "117ba1e7ab06f1c351145232df7e0092"
server_key_iv: "78343edbe4030c09bc3013f5fd236e12"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:02.787580 32442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:02.843724 32442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:02.849125 32442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:02.849534 32442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:02.857200 32442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:26:02.865751 32442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:26:02.865916 32442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260812 11:26:02.866111 32442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:26:02.866225 32442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:02.979296 32442 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:38543
I20260812 11:26:02.979336 32693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:38543 every 8 connection(s)
I20260812 11:26:02.981091 32442 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:26:02.985361 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 32442
I20260812 11:26:02.985553 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:26:02.986132 12731 external_mini_cluster.cc:1428] Setting key 3b518bcd812cdbe97b3e7818f5542ab8
Aug 12 11:26:03 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533962, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
I20260812 11:26:03.034852 32698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:02.984106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:56687 (local address 127.12.110.254:40559)
0812 11:26:03.017350 (+ 33244us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.017355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.017379 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:26:03.018116 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.018121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.018234 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.018374 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:03.019952 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.020704 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.022305 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.022799 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.022979 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.027577 (+ 4598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:03.027586 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:03.027593 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:03.027641 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:03.031357 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:03.032281 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:03.032286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:03.032290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:03.032437 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:03.032931 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:03.032935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:03.032938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:03.033300 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:03.033442 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.034237 (+ 795us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.034466 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33126,"thread_start_us":33245,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
I20260812 11:26:03.035969 32699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:02.984683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40559 (local address 127.12.110.194:56687)
0812 11:26:03.011515 (+ 26832us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:03.011584 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:03.017812 (+ 6228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:03.018662 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:03.018671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:03.019169 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:26:03.019763 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.019780 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.020884 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.020890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.022152 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.022165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.023199 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.023206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.023397 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.024226 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:26:03.024250 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:26:03.027317 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:26:03.031531 (+ 4214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:03.031540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:03.031557 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:03.032139 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:03.032566 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:03.032570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:03.032574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:03.032802 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:03.033442 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:26:03.033448 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:26:03.033860 (+ 412us) client_negotiation.cc:769] Sending connection context
0812 11:26:03.034097 (+ 237us) client_negotiation.cc:240] Negotiation successful
0812 11:26:03.034507 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26359,"thread_start_us":29756,"threads_started":1}
I20260812 11:26:03.039685 32694 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40559
I20260812 11:26:03.039994 32694 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:03.040730 32694 heartbeater.cc:511] Master 127.12.110.254:40559 requested a full tablet report, sending...
I20260812 11:26:03.042294 32042 ts_manager.cc:194] Registered new tserver with Master: c95a2e242ffc46d89c2989081ada6fac (127.12.110.194:38543)
I20260812 11:26:03.043401 32042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:56687
I20260812 11:26:03.055059 32431 heartbeater.cc:503] Master 127.12.110.254:40559 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:26:03.077337 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:41403/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40559
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:26:03.481199 32705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:03.481488 32705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:03.481638 32705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:03.489476 32705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:03.489614 32705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:03.489820 32705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:26:03.501369 32705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.110.212:41403/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:03.503347 32705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:03.505039 32705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:03.521004 32717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:03.521211 32721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:03.521211 32725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:03.522082 32705 server_base.cc:1034] running on GCE node
I20260812 11:26:03.522874 32705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:03.524052 32705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:03.525337 32705 hybrid_clock.cc:648] HybridClock initialized: now 1786533963525295 us; error 35 us; skew 500 ppm
I20260812 11:26:03.525705 32705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:03 dist-test-slave-h6n0 krb5kdc[31163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533963, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:03.529209 32705 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:26:03.531145 32705 webserver.cc:460] Webserver started at http://127.12.110.195:44043/ using document root <none> and password file <none>
I20260812 11:26:03.531893 32705 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:03.532054 32705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:03.532390 32705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:03 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533963, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:03.549701 32705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "39039b7952104baa8284a539effe26ee"
format_stamp: "Formatted at 2026-08-12 11:26:03 on dist-test-slave-h6n0"
server_key: "0a376a42a1c23a50cbc198ee1e2c69f5"
server_key_iv: "f9e199cdbdb1778e04b61f00cd1a7085"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:03.550765 32705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "39039b7952104baa8284a539effe26ee"
format_stamp: "Formatted at 2026-08-12 11:26:03 on dist-test-slave-h6n0"
server_key: "0a376a42a1c23a50cbc198ee1e2c69f5"
server_key_iv: "f9e199cdbdb1778e04b61f00cd1a7085"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:03.556944 32705 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260812 11:26:03.575073 32744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:03.576697 32705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:26:03.576900 32705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "39039b7952104baa8284a539effe26ee"
format_stamp: "Formatted at 2026-08-12 11:26:03 on dist-test-slave-h6n0"
server_key: "0a376a42a1c23a50cbc198ee1e2c69f5"
server_key_iv: "f9e199cdbdb1778e04b61f00cd1a7085"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:03.577162 32705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:03.594761 32705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:03.599255 32705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:03.599589 32705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:03.600664 32705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:26:03.602109 32705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:26:03.602186 32705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:03.602357 32705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:26:03.602415 32705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:03.640766 32705 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:45079
I20260812 11:26:03.640798 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:45079 every 8 connection(s)
I20260812 11:26:03.642130 32705 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:26:03.645314 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 32705
I20260812 11:26:03.645505 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:26:03.646222 12731 external_mini_cluster.cc:1428] Setting key 201d40688be8107ae1ebb2c4340643df
Aug 12 11:26:03 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533963, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:03.665051 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.644821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:39333 (local address 127.12.110.254:40559)
0812 11:26:03.645338 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.645342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.646572 (+ 1230us) server_negotiation.cc:407] Connection header received
0812 11:26:03.649006 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.649010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.649102 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.649197 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:03.650889 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.651642 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.653435 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.653893 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.653999 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.657526 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:03.657531 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:03.657536 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:03.657586 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:03.661944 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:03.662953 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:03.662958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:03.662960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:03.663089 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:03.663533 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:03.663536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:03.663538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:03.663789 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:03.663899 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.664622 (+ 723us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.664792 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260812 11:26:03.666687 485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.645400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:40559 (local address 127.12.110.195:39333)
0812 11:26:03.646375 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:03.646444 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:03.648659 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:03.649494 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:03.649505 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:03.650030 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:26:03.650691 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.650710 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.651929 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.651935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.653288 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.653302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.654129 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.654135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.654342 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.655142 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:26:03.655165 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:26:03.657310 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:26:03.662108 (+ 4798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:03.662119 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:03.662134 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:03.662811 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:03.663201 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:03.663204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:03.663207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:03.663414 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:03.663940 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:26:03.663946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:26:03.664239 (+ 293us) client_negotiation.cc:769] Sending connection context
0812 11:26:03.664484 (+ 245us) client_negotiation.cc:240] Negotiation successful
0812 11:26:03.664861 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":471,"threads_started":1}
I20260812 11:26:03.668251 475 heartbeater.cc:347] Connected to a master server at 127.12.110.254:40559
I20260812 11:26:03.668603 475 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:03.669389 475 heartbeater.cc:511] Master 127.12.110.254:40559 requested a full tablet report, sending...
I20260812 11:26:03.671029 32042 ts_manager.cc:194] Registered new tserver with Master: 39039b7952104baa8284a539effe26ee (127.12.110.195:45079)
I20260812 11:26:03.672032 32042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:39333
I20260812 11:26:03.675244 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:26:03.694221 481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.680070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.12.110.254:40559)
0812 11:26:03.680259 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.680267 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.680733 (+ 466us) server_negotiation.cc:407] Connection header received
0812 11:26:03.680948 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.680952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.681033 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.681150 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:03.682662 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.683523 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.684956 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.685386 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.685533 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.688446 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:03.688451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:03.688455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:03.688488 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:03.691412 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:03.692139 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:03.692151 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:03.692153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:03.692276 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:03.692664 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:03.692669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:03.692672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:03.692964 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:03.693227 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.693598 (+ 371us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.693900 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:26:03.698710 32042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:26:03.706753 32042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:26:03.730247 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.719766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.12.110.194:38543)
0812 11:26:03.720326 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.720337 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.720987 (+ 650us) server_negotiation.cc:407] Connection header received
0812 11:26:03.721306 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.721315 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.721585 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.721789 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:26:03.723785 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.724956 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.726984 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.728531 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.728689 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.728893 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.729666 (+ 773us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.729858 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
I20260812 11:26:03.730263 500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.719427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:38543 (local address 127.0.0.1:43200)
0812 11:26:03.720832 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:03.720878 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:03.721065 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:03.723084 (+ 2019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:03.723091 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:03.723121 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:26:03.723592 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.723608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.725133 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.725137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.726784 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.726807 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.729080 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.729088 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.729332 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.729397 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 11:26:03.729635 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 11:26:03.729878 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":1060,"threads_started":1}
I20260812 11:26:03.734277 32575 tablet_service.cc:1467] Processing CreateTablet for tablet dd48e66aacd74b7893cd16d3ee57798e (DEFAULT_TABLE table=test-table [id=731e4856f828449abfed966457deafa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:26:03.735874 32575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd48e66aacd74b7893cd16d3ee57798e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:26:03.737406 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.721612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.12.110.195:45079)
0812 11:26:03.727893 (+ 6281us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.727904 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.727940 (+ 36us) server_negotiation.cc:407] Connection header received
0812 11:26:03.728050 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.728075 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.728320 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.728559 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:26:03.729761 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.731481 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.734436 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.735943 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.736134 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.736307 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.736888 (+ 581us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.737043 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6078,"thread_start_us":5957,"threads_started":1}
I20260812 11:26:03.737430 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.721272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45079 (local address 127.0.0.1:44216)
0812 11:26:03.721851 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:03.721872 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:03.721983 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:03.728533 (+ 6550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:03.728537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:03.728564 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:26:03.728964 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.728978 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.731653 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.731657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.734255 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.734270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.736495 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.736502 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.736702 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.736726 (+ 24us) client_negotiation.cc:769] Sending connection context
0812 11:26:03.736839 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:26:03.736970 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":144,"thread_start_us":319,"threads_started":1}
I20260812 11:26:03.737576 510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.721923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:39039 (local address 127.0.0.1:33626)
0812 11:26:03.722414 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:03.722427 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:03.722507 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:03.728957 (+ 6450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:03.728961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:03.728994 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:26:03.729442 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.729452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.732055 (+ 2603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.732066 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.734590 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.734603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.736651 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.736658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.736848 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.736890 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:26:03.736984 (+ 94us) client_negotiation.cc:240] Negotiation successful
0812 11:26:03.737120 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"mutex_wait_us":56,"thread_start_us":302,"threads_started":1}
I20260812 11:26:03.738305 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.722343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.12.110.193:39039)
0812 11:26:03.728368 (+ 6025us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.728374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.728394 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:26:03.728495 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.728503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.728718 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.728949 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:26:03.729714 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.731883 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.734745 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.735956 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.736144 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.736320 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.737784 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.737974 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5854,"thread_start_us":5681,"threads_started":1}
I20260812 11:26:03.741524 357 tablet_service.cc:1467] Processing CreateTablet for tablet dd48e66aacd74b7893cd16d3ee57798e (DEFAULT_TABLE table=test-table [id=731e4856f828449abfed966457deafa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:26:03.741571 32309 tablet_service.cc:1467] Processing CreateTablet for tablet dd48e66aacd74b7893cd16d3ee57798e (DEFAULT_TABLE table=test-table [id=731e4856f828449abfed966457deafa8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:26:03.742950 32309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd48e66aacd74b7893cd16d3ee57798e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:26:03.743008 357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd48e66aacd74b7893cd16d3ee57798e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:26:03.751145 515 tablet_bootstrap.cc:492] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Bootstrap starting.
I20260812 11:26:03.754688 517 tablet_bootstrap.cc:492] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Bootstrap starting.
I20260812 11:26:03.755172 515 tablet_bootstrap.cc:654] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Neither blocks nor log segments found. Creating new log.
I20260812 11:26:03.756296 515 log.cc:824] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Log is configured to *not* fsync() on all Append() calls
I20260812 11:26:03.758651 517 tablet_bootstrap.cc:654] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Neither blocks nor log segments found. Creating new log.
I20260812 11:26:03.759611 515 tablet_bootstrap.cc:492] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: No bootstrap required, opened a new log
I20260812 11:26:03.759764 517 log.cc:824] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:26:03.759923 515 ts_tablet_manager.cc:1397] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260812 11:26:03.761197 518 tablet_bootstrap.cc:492] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Bootstrap starting.
I20260812 11:26:03.762873 517 tablet_bootstrap.cc:492] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: No bootstrap required, opened a new log
I20260812 11:26:03.763123 517 ts_tablet_manager.cc:1397] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260812 11:26:03.764047 518 tablet_bootstrap.cc:654] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Neither blocks nor log segments found. Creating new log.
I20260812 11:26:03.764979 518 log.cc:824] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Log is configured to *not* fsync() on all Append() calls
I20260812 11:26:03.767623 518 tablet_bootstrap.cc:492] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: No bootstrap required, opened a new log
I20260812 11:26:03.767880 518 ts_tablet_manager.cc:1397] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260812 11:26:03.770677 515 raft_consensus.cc:348] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.771288 515 raft_consensus.cc:374] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:26:03.771435 515 raft_consensus.cc:729] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c95a2e242ffc46d89c2989081ada6fac, State: Initialized, Role: FOLLOWER
I20260812 11:26:03.772171 515 consensus_queue.cc:260] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.773341 32694 heartbeater.cc:503] Master 127.12.110.254:40559 was elected leader, sending a full tablet report...
I20260812 11:26:03.773654 517 raft_consensus.cc:348] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.774076 517 raft_consensus.cc:374] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:26:03.774183 517 raft_consensus.cc:729] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 778764dad1fc4ba89318ee8df1ccef3e, State: Initialized, Role: FOLLOWER
I20260812 11:26:03.774765 517 consensus_queue.cc:260] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.775761 515 ts_tablet_manager.cc:1428] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260812 11:26:03.776568 518 raft_consensus.cc:348] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.777087 518 raft_consensus.cc:374] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:26:03.777235 518 raft_consensus.cc:729] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39039b7952104baa8284a539effe26ee, State: Initialized, Role: FOLLOWER
I20260812 11:26:03.777499 517 ts_tablet_manager.cc:1428] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260812 11:26:03.777868 518 consensus_queue.cc:260] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.779696 475 heartbeater.cc:503] Master 127.12.110.254:40559 was elected leader, sending a full tablet report...
I20260812 11:26:03.779800 518 ts_tablet_manager.cc:1428] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260812 11:26:03.893973 479 tablet.cc:2369] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:26:03.942729 527 raft_consensus.cc:482] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:26:03.943105 527 raft_consensus.cc:504] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.944931 527 leader_election.cc:288] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c95a2e242ffc46d89c2989081ada6fac (127.12.110.194:38543), 39039b7952104baa8284a539effe26ee (127.12.110.195:45079)
I20260812 11:26:03.952471 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.945459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:57875 (local address 127.12.110.194:38543)
0812 11:26:03.945698 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.945703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.946228 (+ 525us) server_negotiation.cc:407] Connection header received
0812 11:26:03.946490 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.946496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.946620 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.946733 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:26:03.947468 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.948468 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.950472 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.951424 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.951541 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.951667 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.952036 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.952173 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 11:26:03.952527 529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.945346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:38543 (local address 127.12.110.193:57875)
0812 11:26:03.946087 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:03.946109 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:03.946262 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:03.946854 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:03.946861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:03.946898 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:26:03.947284 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.947298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.948611 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.948617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.950313 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.950329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.951651 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.951659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.951860 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.951896 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:26:03.951996 (+ 100us) client_negotiation.cc:240] Negotiation successful
0812 11:26:03.952116 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":20,"thread_start_us":414,"threads_started":1}
I20260812 11:26:03.952749 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.946490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:38947 (local address 127.12.110.195:45079)
0812 11:26:03.946734 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:03.946740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:03.947038 (+ 298us) server_negotiation.cc:407] Connection header received
0812 11:26:03.947187 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:03.947195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:03.947276 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:03.947386 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:26:03.948095 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.948948 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.950862 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.951748 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.951924 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.952016 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:03.952366 (+ 350us) server_negotiation.cc:299] Negotiation successful
0812 11:26:03.952503 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 11:26:03.952955 531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:03.946254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45079 (local address 127.12.110.193:38947)
0812 11:26:03.946931 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:03.946945 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:03.947060 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:03.947468 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:03.947473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:03.947501 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:26:03.947920 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.947932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.949116 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.949119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.950716 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:03.950730 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:03.952015 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:03.952022 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:03.952187 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:03.952222 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:26:03.952321 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:26:03.952439 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":20,"thread_start_us":393,"threads_started":1}
I20260812 11:26:03.953804 387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48e66aacd74b7893cd16d3ee57798e" candidate_uuid: "778764dad1fc4ba89318ee8df1ccef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39039b7952104baa8284a539effe26ee" is_pre_election: true
I20260812 11:26:03.953845 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48e66aacd74b7893cd16d3ee57798e" candidate_uuid: "778764dad1fc4ba89318ee8df1ccef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95a2e242ffc46d89c2989081ada6fac" is_pre_election: true
I20260812 11:26:03.954340 387 raft_consensus.cc:2454] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778764dad1fc4ba89318ee8df1ccef3e in term 0.
I20260812 11:26:03.954376 32615 raft_consensus.cc:2454] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 778764dad1fc4ba89318ee8df1ccef3e in term 0.
I20260812 11:26:03.955066 32226 leader_election.cc:302] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39039b7952104baa8284a539effe26ee, 778764dad1fc4ba89318ee8df1ccef3e; no voters:
I20260812 11:26:03.955462 527 raft_consensus.cc:2781] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:26:03.955617 527 raft_consensus.cc:482] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:26:03.955773 527 raft_consensus.cc:3037] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:26:03.958244 527 raft_consensus.cc:504] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.958997 527 leader_election.cc:288] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [CANDIDATE]: Term 1 election: Requested vote from peers c95a2e242ffc46d89c2989081ada6fac (127.12.110.194:38543), 39039b7952104baa8284a539effe26ee (127.12.110.195:45079)
I20260812 11:26:03.959436 32615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48e66aacd74b7893cd16d3ee57798e" candidate_uuid: "778764dad1fc4ba89318ee8df1ccef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c95a2e242ffc46d89c2989081ada6fac"
I20260812 11:26:03.959533 387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd48e66aacd74b7893cd16d3ee57798e" candidate_uuid: "778764dad1fc4ba89318ee8df1ccef3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39039b7952104baa8284a539effe26ee"
I20260812 11:26:03.959684 32615 raft_consensus.cc:3037] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:26:03.959750 387 raft_consensus.cc:3037] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:26:03.961906 32615 raft_consensus.cc:2454] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778764dad1fc4ba89318ee8df1ccef3e in term 1.
I20260812 11:26:03.962001 387 raft_consensus.cc:2454] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 778764dad1fc4ba89318ee8df1ccef3e in term 1.
I20260812 11:26:03.962330 32226 leader_election.cc:302] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 778764dad1fc4ba89318ee8df1ccef3e, c95a2e242ffc46d89c2989081ada6fac; no voters:
I20260812 11:26:03.962647 527 raft_consensus.cc:2781] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:26:03.963115 527 raft_consensus.cc:686] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 1 LEADER]: Becoming Leader. State: Replica: 778764dad1fc4ba89318ee8df1ccef3e, State: Running, Role: LEADER
I20260812 11:26:03.963604 527 consensus_queue.cc:237] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } }
I20260812 11:26:03.968286 32038 catalog_manager.cc:5495] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 778764dad1fc4ba89318ee8df1ccef3e (127.12.110.193). New cstate: current_term: 1 leader_uuid: "778764dad1fc4ba89318ee8df1ccef3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "778764dad1fc4ba89318ee8df1ccef3e" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } health_report { overall_health: HEALTHY } } }
W20260812 11:26:04.010232 32432 tablet.cc:2369] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:26:04.017802 32695 tablet.cc:2369] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:26:04.029803 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:04.024276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.12.110.193:39039)
0812 11:26:04.024554 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:04.024568 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:04.024590 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:26:04.024699 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:04.024702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:04.024777 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:04.024934 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:26:04.025368 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.026123 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.027649 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.028047 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.028202 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:04.028690 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:26:04.029183 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:26:04.029342 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:04.029456 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 11:26:04.029546 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:26:04.038295 387 raft_consensus.cc:1261] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 1 FOLLOWER]: Refusing update from remote peer 778764dad1fc4ba89318ee8df1ccef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:26:04.038441 32615 raft_consensus.cc:1261] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 1 FOLLOWER]: Refusing update from remote peer 778764dad1fc4ba89318ee8df1ccef3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:26:04.039418 534 consensus_queue.cc:1035] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39039b7952104baa8284a539effe26ee" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:26:04.039819 527 consensus_queue.cc:1035] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c95a2e242ffc46d89c2989081ada6fac" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:26:04.051455 544 mvcc.cc:205] Tried to move back new op lower bound from 7317643116690591744 to 7317643116398473216. Current Snapshot: MvccSnapshot[applied={T|T < 7317643116690591744}]
I20260812 11:26:04.052971 547 mvcc.cc:205] Tried to move back new op lower bound from 7317643116690591744 to 7317643116398473216. Current Snapshot: MvccSnapshot[applied={T|T < 7317643116690591744}]
I20260812 11:26:04.068266 32038 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:26:04.068550 32038 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:26:04.072446 32038 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45593998800b47d39fa09f268f8aa675: Sending DeleteTablet for 3 replicas of tablet dd48e66aacd74b7893cd16d3ee57798e
I20260812 11:26:04.073632 32575 tablet_service.cc:1514] Processing DeleteTablet for tablet dd48e66aacd74b7893cd16d3ee57798e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:26:04 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:43200
I20260812 11:26:04.073746 32309 tablet_service.cc:1514] Processing DeleteTablet for tablet dd48e66aacd74b7893cd16d3ee57798e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:26:04 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:33626
I20260812 11:26:04.073802 357 tablet_service.cc:1514] Processing DeleteTablet for tablet dd48e66aacd74b7893cd16d3ee57798e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:26:04 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:44216
I20260812 11:26:04.075723 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 32193
I20260812 11:26:04.085186 559 tablet_replica.cc:330] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: stopping tablet replica
I20260812 11:26:04.084455 560 tablet_replica.cc:330] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: stopping tablet replica
I20260812 11:26:04.085901 559 raft_consensus.cc:2229] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:26:04.085906 560 raft_consensus.cc:2229] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:26:04.086603 559 raft_consensus.cc:2258] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:26:04.086647 560 raft_consensus.cc:2258] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:26:04.088558 559 ts_tablet_manager.cc:1905] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:26:04.090766 560 ts_tablet_manager.cc:1905] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:26:04.093329 561 tablet_replica.cc:330] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: stopping tablet replica
I20260812 11:26:04.093884 561 raft_consensus.cc:2229] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:26:04.094620 561 raft_consensus.cc:2258] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 12 11:26:04 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:26:04.099551 559 ts_tablet_manager.cc:1918] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:26:04.099747 559 log.cc:1196] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/dd48e66aacd74b7893cd16d3ee57798e
I20260812 11:26:04.100353 559 ts_tablet_manager.cc:1939] T dd48e66aacd74b7893cd16d3ee57798e P c95a2e242ffc46d89c2989081ada6fac: Deleting consensus metadata
I20260812 11:26:04.101549 561 ts_tablet_manager.cc:1905] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:26:04.102392 560 ts_tablet_manager.cc:1918] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:26:04.102382 32020 catalog_manager.cc:4841] TS c95a2e242ffc46d89c2989081ada6fac (127.12.110.194:38543): tablet dd48e66aacd74b7893cd16d3ee57798e (table test-table [id=731e4856f828449abfed966457deafa8]) successfully deleted
I20260812 11:26:04.102685 560 log.cc:1196] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/dd48e66aacd74b7893cd16d3ee57798e
I20260812 11:26:04.103384 560 ts_tablet_manager.cc:1939] T dd48e66aacd74b7893cd16d3ee57798e P 778764dad1fc4ba89318ee8df1ccef3e: Deleting consensus metadata
I20260812 11:26:04.105409 32020 catalog_manager.cc:4841] TS 778764dad1fc4ba89318ee8df1ccef3e (127.12.110.193:39039): tablet dd48e66aacd74b7893cd16d3ee57798e (table test-table [id=731e4856f828449abfed966457deafa8]) successfully deleted
I20260812 11:26:04.107961 561 ts_tablet_manager.cc:1918] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:26:04.108182 561 log.cc:1196] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/dd48e66aacd74b7893cd16d3ee57798e
I20260812 11:26:04.108654 561 ts_tablet_manager.cc:1939] T dd48e66aacd74b7893cd16d3ee57798e P 39039b7952104baa8284a539effe26ee: Deleting consensus metadata
I20260812 11:26:04.108723 513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:04.076718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.12.110.193:39039)
0812 11:26:04.076989 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:04.077000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:04.084544 (+ 7544us) server_negotiation.cc:407] Connection header received
0812 11:26:04.084709 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:04.084719 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:04.084873 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:04.085034 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:04.087000 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.088248 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.090347 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.091153 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.091310 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:04.096988 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:04.097008 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:04.097026 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:04.097085 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:04.103813 (+ 6728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:04.105693 (+ 1880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:04.105704 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:04.105707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:04.105954 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:04.106606 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:04.106612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:04.106615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:04.107102 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:04.107390 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:04.108037 (+ 647us) server_negotiation.cc:299] Negotiation successful
0812 11:26:04.108314 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 11:26:04.110335 32423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:04.110631 32020 catalog_manager.cc:4841] TS 39039b7952104baa8284a539effe26ee (127.12.110.195:45079): tablet dd48e66aacd74b7893cd16d3ee57798e (table test-table [id=731e4856f828449abfed966457deafa8]) successfully deleted
I20260812 11:26:04.209893 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 32193
I20260812 11:26:04.236596 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 32442
Aug 12 11:26:04 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:26:04.252810 501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:04.237389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43202 (local address 127.12.110.194:38543)
0812 11:26:04.237555 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:04.237560 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:04.237578 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:26:04.237739 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:04.237743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:04.237828 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:04.237961 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:04.239420 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.240138 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.241437 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.241837 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.241978 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:04.246655 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:04.246670 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:04.246686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:04.246722 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:04.249664 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:04.250437 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:04.250445 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:04.250447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:04.250767 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:04.251213 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:04.251219 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:04.251222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:04.251592 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:04.251909 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:04.252211 (+ 302us) server_negotiation.cc:299] Negotiation successful
0812 11:26:04.252432 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:26:04.253458 32689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:04.356271 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 32442
I20260812 11:26:04.382988 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 32705
Aug 12 11:26:04 dist-test-slave-h6n0 krb5kdc[31163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:26:04.399410 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:04.383773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44228 (local address 127.12.110.195:45079)
0812 11:26:04.383960 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:04.383964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:04.383982 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:26:04.384096 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:04.384101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:04.384174 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:04.384261 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:04.385769 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.386622 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.387908 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:04.388340 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:04.388449 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:04.393931 (+ 5482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:04.393944 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:04.393961 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:04.393995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:04.396771 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:04.397420 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:04.397427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:04.397429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:04.397570 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:04.397898 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:04.397902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:04.397904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:04.398255 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:04.398470 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:04.398848 (+ 378us) server_negotiation.cc:299] Negotiation successful
0812 11:26:04.399014 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:26:04.400343 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:04.491349 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 32705
I20260812 11:26:04.519560 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 31990
I20260812 11:26:04.520833 32098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:04.634109 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 31990
I20260812 11:26:04.670076 12731 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 11:26:05.016031 12731 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 11:26:05.016309 12731 mini_ranger.cc:66] Stopping Ranger...
I20260812 11:26:05.371855 12731 mini_ranger.cc:68] Stopped Ranger
2026-08-12 11:26:05.372 UTC [31222] LOG: received smart shutdown request
2026-08-12 11:26:05.376 UTC [31222] LOG: background worker "logical replication launcher" (PID 31238) exited with exit code 1
2026-08-12 11:26:05.376 UTC [31228] LOG: shutting down
2026-08-12 11:26:05.546 UTC [31222] LOG: database system is shut down
2026-08-12T11:26:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53894 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:26:05 dist-test-slave-h6n0 krb5kdc[588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:26:05 dist-test-slave-h6n0 krb5kdc[588](info): set up 2 sockets
Aug 12 11:26:05 dist-test-slave-h6n0 krb5kdc[588](info): commencing operation
krb5kdc: starting...
W20260812 11:26:07.108906 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.374s user 0.003s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:26:07 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786533967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:26:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:26:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/rangeradmin_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/rangeradmin_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/rangerlookup_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/rangerlookup_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
I20260812 11:26:07.538769 12731 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasklUqj9l/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/postgres -l logfile start
2026-08-12 11:26:10.216 UTC [644] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-12 11:26:10.216 UTC [644] LOG: listening on IPv4 address "127.12.110.212", port 50305
2026-08-12 11:26:10.221 UTC [644] LOG: listening on Unix socket "/tmp/.s.PGSQL.50305"
2026-08-12 11:26:10.248 UTC [649] LOG: database system was shut down at 2026-08-12 11:26:08 UTC
2026-08-12 11:26:10.251 UTC [644] LOG: database system is ready to accept connections
I20260812 11:26:11.495560 12731 mini_postgres.cc:96] Postgres bound to 50305
2026-08-12 11:26:11.522 UTC [661] FATAL: database "slave" does not exist
127.12.110.212:50305 - accepting connections
I20260812 11:26:11.523397 12731 mini_ranger.cc:161] Starting Ranger...
I20260812 11:26:11.554943 12731 mini_ranger.cc:84] Created miniranger Postgres user
I20260812 11:26:12.055809 12731 mini_ranger.cc:87] Created ranger Postgres database
I20260812 11:26:12.055944 12731 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-admin
2026-08-12 11:26:12,378 [I] DB FLAVOR :POSTGRES
2026-08-12 11:26:12,379 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:26:12,379 [I] Checking connection..
2026-08-12 11:26:12,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-12 11:26:12,640 [I] Checking connection passed.
2026-08-12 11:26:12,640 [I] --------- Verifying version history table ---------
2026-08-12 11:26:12,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:26:12,910 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:26:12,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:26:13,182 [I] Table x_db_version_h does not exist in database ranger
2026-08-12 11:26:13,183 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-12 11:26:13,183 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-12 11:26:13.489 UTC [787] WARNING: there is no transaction in progress
2026-08-12 11:26:13,499 [I] create_dbversion_catalog.sql file imported successfully
2026-08-12 11:26:13,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-12 11:26:13,778 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-12 11:26:13,778 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:26:13,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-12 11:26:14,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-12 11:26:14,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h6n0.c.gcp-upstream.internal','N') ;"
2026-08-12 11:26:14,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:26:14,860 [I] Table x_portal_user does not exist in database ranger
2026-08-12 11:26:14,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:26:15,130 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-12 11:26:15,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:26:15,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:26:15,635 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-12 11:26:15,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-12 11:26:16.637 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.655 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.672 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.687 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.703 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.721 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.783 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.790 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.803 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.814 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.823 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.836 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.845 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.851 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.857 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:16.864 UTC [980] WARNING: there is no transaction in progress
2026-08-12 11:26:17,510 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-12 11:26:17,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-12 11:26:17,768 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-12 11:26:17,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-12 11:26:18,049 [I] Table x_portal_user already exists in database 'ranger'
2026-08-12 11:26:18,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-12 11:26:18,336 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-12 11:26:18,337 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-12 11:26:18,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-12 11:26:18,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h6n0.c.gcp-upstream.internal';"
2026-08-12 11:26:19,132 [I] CORE_DB_SCHEMA import status has been updated
2026-08-12 11:26:19,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-12 11:26:19,387 [I] DB_PATCHES have already been applied
I20260812 11:26:19.394299 12731 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-admin:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasklUqj9l/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260812 11:26:19.394390 12731 mini_ranger.cc:193] Using host: 127.12.110.212
I20260812 11:26:19.416893 12731 mini_ranger.cc:227] Ranger admin URL: http://127.12.110.212:33619
Aug 12, 2026 11:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp
Aug 12, 2026 11:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 12, 2026 11:26:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 12, 2026 11:26:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-admin/ews/webapp].
Aug 12, 2026 11:26:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260812 11:26:20.907291 12731 mini_ranger.cc:160] Time spent starting Ranger: real 9.384s user 0.004s sys 0.066s
Aug 12, 2026 11:26:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 12, 2026 11:26:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 12, 2026 11:26:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 12, 2026 11:26:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 12, 2026 11:26:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260812 11:26:44.927876 12731 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.12.110.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.12.110.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/rangerkms_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.12.110.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/rangerkms_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.12.110.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.12.110.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.12.110.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/HTTP_127.12.110.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 12 11:26:45 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534005, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260812 11:26:45.137735 12731 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260812 11:26:45.166572 12731 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260812 11:26:45.991238 12731 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260812 11:26:45.991369 12731 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms
I20260812 11:26:45.991426 12731 mini_ranger_kms.cc:227] Using postgres at 127.12.110.212:50305
2026-08-12 11:26:46,380 [I] DB FLAVOR :POSTGRES
2026-08-12 11:26:46,380 [I] --------- Verifying Ranger DB connection ---------
2026-08-12 11:26:46,380 [I] Checking connection
2026-08-12 11:26:46,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-12 11:26:46,655 [I] connection success
2026-08-12 11:26:46,655 [I] --------- Verifying Ranger DB tables ---------
2026-08-12 11:26:46,656 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-12 11:26:46,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-12 11:26:46,895 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-12 11:26:46,895 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-12 11:26:46,896 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-12 11:26:46,896 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.12.110.212:50305/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-12 11:26:47,163 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260812 11:26:47.668330 12731 mini_ranger_kms.cc:310] Created kms service
I20260812 11:26:47.902870 12731 mini_ranger_kms.cc:326] Created kudu user
I20260812 11:26:47.948509 12731 mini_ranger_kms.cc:343] Created rangerkms user
Aug 12 11:26:47 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534005, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:48.384361 12731 mini_ranger_kms.cc:384] Added ranger policy
I20260812 11:26:48.384725 12731 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms:/tmp/dist-test-tasklUqj9l/build/asan/bin/postgresql.jar:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasklUqj9l/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260812 11:26:48.384797 12731 mini_ranger_kms.cc:240] Using host: 127.12.110.212
I20260812 11:26:48.414990 12731 mini_ranger_kms.cc:276] Ranger KMS PID: 1309
I20260812 11:26:48.415243 12731 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.12.110.212:39943
Aug 12, 2026 11:26:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 12, 2026 11:26:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 12, 2026 11:26:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 12, 2026 11:26:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 12, 2026 11:26:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-kms/ews/webapp].
Aug 12, 2026 11:26:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 12 11:26:50 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786534010, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.110.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 12 11:26:51 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534010, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.110.212@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:51.349676 12731 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.212s user 0.068s sys 0.358s
I20260812 11:26:51.350024 12731 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260812 11:26:51.350368 12731 mini_ranger_kms.cc:401] 127.12.110.212:39943/kms/v1/keys
Aug 12 11:26:51 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534010, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.12.110.212@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
Aug 12 11:26:53 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:26:53.914568 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43879
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:43879
--ranger_config_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:26:54.261713 1395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:54.262037 1395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:54.262203 1395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:54.269515 1395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:26:54.269629 1395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:54.269721 1395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:54.269971 1395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:26:54.270202 1395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:26:54.284188 1395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
--ranger_kms_url=127.12.110.212:39943/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:43879
--ranger_config_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:54.285586 1395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:54.287657 1395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:54.302167 1406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:54.302203 1403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:54.302260 1408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:54.303053 1395 server_base.cc:1034] running on GCE node
I20260812 11:26:54.303998 1395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:54.305536 1395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:54.306749 1395 hybrid_clock.cc:648] HybridClock initialized: now 1786534014306694 us; error 41 us; skew 500 ppm
I20260812 11:26:54.307201 1395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:54 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534014, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:54.310920 1395 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:26:54.312765 1395 webserver.cc:460] Webserver started at http://127.12.110.254:34025/ using document root <none> and password file <none>
I20260812 11:26:54.313455 1395 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:54.313591 1395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:54.314033 1395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:54 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534014, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:54.375960 1395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "40fffcc3251a40fa9c96671c588a1d45"
format_stamp: "Formatted at 2026-08-12 11:26:54 on dist-test-slave-h6n0"
server_key: "4021a82e0da78699ee88ba793b4a0cd1"
server_key_iv: "5d2b0f90a9f1d9a3fc3d4e1d50033a3f"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:54.376978 1395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "40fffcc3251a40fa9c96671c588a1d45"
format_stamp: "Formatted at 2026-08-12 11:26:54 on dist-test-slave-h6n0"
server_key: "4021a82e0da78699ee88ba793b4a0cd1"
server_key_iv: "5d2b0f90a9f1d9a3fc3d4e1d50033a3f"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:54.382882 1395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:26:54.403007 1420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:54.405431 1395 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:26:54.405680 1395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "40fffcc3251a40fa9c96671c588a1d45"
format_stamp: "Formatted at 2026-08-12 11:26:54 on dist-test-slave-h6n0"
server_key: "4021a82e0da78699ee88ba793b4a0cd1"
server_key_iv: "5d2b0f90a9f1d9a3fc3d4e1d50033a3f"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:54.405954 1395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:54.437918 1395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:54.441773 1395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:54.442044 1395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:54.461668 1395 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:43879
I20260812 11:26:54.461678 1518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:43879 every 8 connection(s)
I20260812 11:26:54.463130 1395 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:26:54.466915 1520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:26:54.468556 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 1395
I20260812 11:26:54.468802 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:26:54.469501 12731 external_mini_cluster.cc:1428] Setting key 6a0b8204278dacb3c4a29053116026fb
I20260812 11:26:54.480590 1520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Bootstrap starting.
I20260812 11:26:54.484560 1520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Neither blocks nor log segments found. Creating new log.
I20260812 11:26:54.485817 1520 log.cc:824] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:26:54 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:54.488775 1520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: No bootstrap required, opened a new log
I20260812 11:26:54.494496 1524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:54.471691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.12.110.254:43879)
0812 11:26:54.472560 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:54.472577 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:54.472634 (+ 57us) server_negotiation.cc:407] Connection header received
0812 11:26:54.475066 (+ 2432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:54.475098 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:54.475576 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:54.476015 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:54.477290 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:54.479352 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:54.481343 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:54.482024 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:54.482168 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:54.487154 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:54.487164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:54.487178 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:54.487221 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:54.490257 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:54.491102 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:54.491109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:54.491115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:54.491282 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:54.491717 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:54.491722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:54.491736 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:54.492270 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:54.492578 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:54.493121 (+ 543us) server_negotiation.cc:299] Negotiation successful
0812 11:26:54.493506 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":395,"threads_started":1}
I20260812 11:26:54.497035 1520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:54.497455 1520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:26:54.497581 1520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fffcc3251a40fa9c96671c588a1d45, State: Initialized, Role: FOLLOWER
I20260812 11:26:54.498209 1520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:54.498454 1520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:26:54.498646 1520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:26:54.498811 1520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:26:54.501252 1520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:54.501708 1520 leader_election.cc:302] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40fffcc3251a40fa9c96671c588a1d45; no voters:
I20260812 11:26:54.502306 1520 leader_election.cc:288] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:26:54.502483 1526 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:26:54.503515 1526 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 LEADER]: Becoming Leader. State: Replica: 40fffcc3251a40fa9c96671c588a1d45, State: Running, Role: LEADER
I20260812 11:26:54.504074 1526 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:54.504696 1520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:26:54.507478 1527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40fffcc3251a40fa9c96671c588a1d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } } }
I20260812 11:26:54.507908 1527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: This master's current role is: LEADER
I20260812 11:26:54.507743 1528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40fffcc3251a40fa9c96671c588a1d45. Latest consensus state: current_term: 1 leader_uuid: "40fffcc3251a40fa9c96671c588a1d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } } }
I20260812 11:26:54.508134 1528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: This master's current role is: LEADER
I20260812 11:26:54.508826 1532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:26:54.515182 1532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:26:54.520740 1520 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260812 11:26:54.535432 1532 catalog_manager.cc:1347] Generated new cluster ID: 1939c77778474ed29926f7c39663b0de
I20260812 11:26:54.535637 1532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:26:54.548468 1532 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:26:54.550819 1532 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:26:54.559047 1532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Generated new TSK 0
I20260812 11:26:54.560066 1532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:26:55.340744 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:43879
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:26:55 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786534015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260812 11:26:55.796032 1592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:55.796393 1592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:55.796577 1592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:55.804149 1592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:55.804304 1592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:55.804527 1592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:26:55.815094 1592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.12.110.212:39943/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:55.816605 1592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:55.817898 1592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:55.838761 1615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:55.841730 1618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:55.846122 1629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:55.846452 1592 server_base.cc:1034] running on GCE node
I20260812 11:26:55.847325 1592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:55.848660 1592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:55.850006 1592 hybrid_clock.cc:648] HybridClock initialized: now 1786534015849923 us; error 72 us; skew 500 ppm
I20260812 11:26:55.850400 1592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:55 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:55.854779 1592 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:26:55.865880 1592 webserver.cc:460] Webserver started at http://127.12.110.193:44675/ using document root <none> and password file <none>
I20260812 11:26:55.866832 1592 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:55.867030 1592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:55.867427 1592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:55 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:55.888447 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "b8df244fc61741e59f885648276269e7"
format_stamp: "Formatted at 2026-08-12 11:26:55 on dist-test-slave-h6n0"
server_key: "144810609ac666786bb5591bd754a770"
server_key_iv: "0fcb31a8a840a69a5861542835fe3e11"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:55.889644 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "b8df244fc61741e59f885648276269e7"
format_stamp: "Formatted at 2026-08-12 11:26:55 on dist-test-slave-h6n0"
server_key: "144810609ac666786bb5591bd754a770"
server_key_iv: "0fcb31a8a840a69a5861542835fe3e11"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:55.908826 1592 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.010s sys 0.007s
I20260812 11:26:55.934098 1639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:55.936069 1592 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.004s
I20260812 11:26:55.936365 1592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "b8df244fc61741e59f885648276269e7"
format_stamp: "Formatted at 2026-08-12 11:26:55 on dist-test-slave-h6n0"
server_key: "144810609ac666786bb5591bd754a770"
server_key_iv: "0fcb31a8a840a69a5861542835fe3e11"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:55.936728 1592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:55.969735 1592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:55.975049 1592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:55.975399 1592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:55.976516 1592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:26:55.978240 1592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:26:55.978348 1592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:55.978490 1592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:26:55.978601 1592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 11:26:56.161825 1592 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:45097
I20260812 11:26:56.161865 1845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:45097 every 8 connection(s)
I20260812 11:26:56.163793 1592 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:26:56.169147 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 1592
I20260812 11:26:56.169405 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:26:56.170512 12731 external_mini_cluster.cc:1428] Setting key 3e623a4ab0ec4c52419f7331fd7e8d5a
Aug 12 11:26:56 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:56.200289 1852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:56.171519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:36639 (local address 127.12.110.254:43879)
0812 11:26:56.175782 (+ 4263us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:56.175790 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:56.175814 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:26:56.176277 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:56.176286 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:56.176387 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:56.176496 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:56.178806 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.180069 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.182619 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.183282 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.183382 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:56.191567 (+ 8185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:56.191573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:56.191576 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:56.191611 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:56.195227 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:56.196219 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:56.196225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:56.196228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:56.196388 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:56.196954 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:56.196960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:56.196963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:56.197327 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:56.197463 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:56.199558 (+ 2095us) server_negotiation.cc:299] Negotiation successful
0812 11:26:56.199833 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3981,"spinlock_wait_cycles":4992,"thread_start_us":3941,"threads_started":1}
I20260812 11:26:56.203493 1850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:56.167591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:43879 (local address 127.12.110.193:36639)
0812 11:26:56.171699 (+ 4108us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:56.171756 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:56.174506 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:56.176960 (+ 2454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:56.176974 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:56.177708 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:26:56.178598 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:56.178626 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.180297 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.180305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:56.182409 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:56.182425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.183735 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.183743 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:56.184032 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:56.184872 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:26:56.184911 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:26:56.191307 (+ 6396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:26:56.195414 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:56.195425 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:56.195443 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:56.196073 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:56.196567 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:56.196572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:56.196576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:56.196815 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:56.197570 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:26:56.197578 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:26:56.198375 (+ 797us) client_negotiation.cc:769] Sending connection context
0812 11:26:56.201280 (+ 2905us) client_negotiation.cc:240] Negotiation successful
0812 11:26:56.202067 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3763,"thread_start_us":3542,"threads_started":1}
I20260812 11:26:56.205637 1848 heartbeater.cc:347] Connected to a master server at 127.12.110.254:43879
I20260812 11:26:56.205941 1848 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:56.206804 1848 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
I20260812 11:26:56.209292 1454 ts_manager.cc:194] Registered new tserver with Master: b8df244fc61741e59f885648276269e7 (127.12.110.193:45097)
I20260812 11:26:56.211544 1454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:36639
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Aug 12 11:26:56 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
W20260812 11:26:56.283815 12731 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.12.110.194: real 0.109s user 0.006s sys 0.021s
I20260812 11:26:56.284786 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:43879
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:26:56 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534015, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
W20260812 11:26:56.788668 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:56.789175 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:56.789371 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:56.800171 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:56.800346 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:56.800571 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:26:56.815317 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.12.110.212:39943/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:56.817270 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:56.819181 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:56.836424 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:56.836427 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:56.843251 1856 server_base.cc:1034] running on GCE node
W20260812 11:26:56.843886 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:56.844740 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:56.845944 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:56.847433 1856 hybrid_clock.cc:648] HybridClock initialized: now 1786534016847300 us; error 56 us; skew 500 ppm
I20260812 11:26:56.848112 1856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:56 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534016, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:56.852614 1856 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:26:56.877883 1856 webserver.cc:460] Webserver started at http://127.12.110.194:40591/ using document root <none> and password file <none>
I20260812 11:26:56.879133 1856 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:56.879339 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:56.879798 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:56 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534016, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:56.903093 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "dd03439b6ded4fb99b5a0df47e19a0bc"
format_stamp: "Formatted at 2026-08-12 11:26:56 on dist-test-slave-h6n0"
server_key: "fe460e4a577d3bf70efdec463b53aa8a"
server_key_iv: "6d78bd5a2cb007d4654928a1f9ac97fc"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:56.904201 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "dd03439b6ded4fb99b5a0df47e19a0bc"
format_stamp: "Formatted at 2026-08-12 11:26:56 on dist-test-slave-h6n0"
server_key: "fe460e4a577d3bf70efdec463b53aa8a"
server_key_iv: "6d78bd5a2cb007d4654928a1f9ac97fc"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:56.909816 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 11:26:56.926921 1902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:56.928689 1856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:26:56.928905 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "dd03439b6ded4fb99b5a0df47e19a0bc"
format_stamp: "Formatted at 2026-08-12 11:26:56 on dist-test-slave-h6n0"
server_key: "fe460e4a577d3bf70efdec463b53aa8a"
server_key_iv: "6d78bd5a2cb007d4654928a1f9ac97fc"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:56.929176 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:56.941886 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:56.947082 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:56.947366 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:56.948333 1856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:26:56.949900 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:26:56.949981 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:56.950067 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:26:56.950114 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:56.984305 1856 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:43913
I20260812 11:26:56.984352 2109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:43913 every 8 connection(s)
I20260812 11:26:56.985601 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:26:56.986886 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 1856
I20260812 11:26:56.987088 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:26:56.987789 12731 external_mini_cluster.cc:1428] Setting key d46c24607d5711dd24d7c66c117980a0
Aug 12 11:26:56 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534016, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:57.007143 2117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:56.988626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:54501 (local address 127.12.110.254:43879)
0812 11:26:56.989187 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:56.989193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:56.990151 (+ 958us) server_negotiation.cc:407] Connection header received
0812 11:26:56.992097 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:56.992103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:56.992191 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:56.992307 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:56.993718 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.994474 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.996097 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.996530 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.996667 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:57.000518 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:57.000523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:57.000527 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:57.000559 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:57.003462 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.004252 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.004261 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.004266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.004459 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.004979 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.004985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.004988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.005291 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:57.005459 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:57.006271 (+ 812us) server_negotiation.cc:299] Negotiation successful
0812 11:26:57.006589 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260812 11:26:57.007800 2118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:56.989159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:43879 (local address 127.12.110.194:54501)
0812 11:26:56.989987 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:56.990043 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:56.991840 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:56.992548 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:56.992556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:56.992988 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:26:56.993548 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:56.993564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.994671 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.994677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:56.995927 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:56.995947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:56.996759 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:56.996768 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:56.997021 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:56.998139 (+ 1118us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:26:56.998162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:26:57.000292 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:26:57.003637 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:57.003643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:57.003656 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:57.004115 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:57.004604 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:57.004607 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:57.004609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:57.004806 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:57.005430 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:26:57.005438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:26:57.005768 (+ 330us) client_negotiation.cc:769] Sending connection context
0812 11:26:57.006116 (+ 348us) client_negotiation.cc:240] Negotiation successful
0812 11:26:57.006506 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":350,"threads_started":1}
I20260812 11:26:57.009363 2114 heartbeater.cc:347] Connected to a master server at 127.12.110.254:43879
I20260812 11:26:57.009686 2114 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:57.010396 2114 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
I20260812 11:26:57.012060 1454 ts_manager.cc:194] Registered new tserver with Master: dd03439b6ded4fb99b5a0df47e19a0bc (127.12.110.194:43913)
I20260812 11:26:57.012938 1454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:54501
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:26:57.063598 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:43879
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:26:57.215574 1848 heartbeater.cc:503] Master 127.12.110.254:43879 was elected leader, sending a full tablet report...
W20260812 11:26:57.430440 2122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:57.430774 2122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:57.430887 2122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:57.438243 2122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:57.438356 2122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:57.438472 2122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:26:57.448490 2122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.12.110.212:39943/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:57.450944 2122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:57.452603 2122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:57.466027 2133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:57.466120 2134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:57.467304 2136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:57.468237 2122 server_base.cc:1034] running on GCE node
I20260812 11:26:57.468959 2122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:57.470253 2122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:57.471508 2122 hybrid_clock.cc:648] HybridClock initialized: now 1786534017471405 us; error 129 us; skew 500 ppm
I20260812 11:26:57.471884 2122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:57 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534017, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:57.475574 2122 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:26:57.477104 2122 webserver.cc:460] Webserver started at http://127.12.110.195:35695/ using document root <none> and password file <none>
I20260812 11:26:57.477849 2122 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:57.477996 2122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:26:57.478328 2122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 12 11:26:57 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534017, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:57.496901 2122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "d69b920832674b68beecdb748b7991ca"
format_stamp: "Formatted at 2026-08-12 11:26:57 on dist-test-slave-h6n0"
server_key: "f058135e88b5e77a200f18c365de5aa8"
server_key_iv: "afcd9bde752eed7b5973729c22ac5fa5"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:57.497833 2122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "d69b920832674b68beecdb748b7991ca"
format_stamp: "Formatted at 2026-08-12 11:26:57 on dist-test-slave-h6n0"
server_key: "f058135e88b5e77a200f18c365de5aa8"
server_key_iv: "afcd9bde752eed7b5973729c22ac5fa5"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:57.504884 2122 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 11:26:57.523434 2149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:57.525529 2122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:26:57.525839 2122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "d69b920832674b68beecdb748b7991ca"
format_stamp: "Formatted at 2026-08-12 11:26:57 on dist-test-slave-h6n0"
server_key: "f058135e88b5e77a200f18c365de5aa8"
server_key_iv: "afcd9bde752eed7b5973729c22ac5fa5"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:57.526118 2122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:57.554399 2122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:57.558272 2122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:57.558631 2122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:57.559633 2122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:26:57.561074 2122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:26:57.561153 2122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:57.561235 2122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:26:57.561283 2122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:57.595937 2122 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:43041
I20260812 11:26:57.595986 2356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:43041 every 8 connection(s)
I20260812 11:26:57.597218 2122 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:26:57.598160 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 2122
I20260812 11:26:57.598351 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:26:57.599010 12731 external_mini_cluster.cc:1428] Setting key da723974a29fcd500a2532e94ff47082
Aug 12 11:26:57 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534017, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:57.618438 2361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:57.600196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:60269 (local address 127.12.110.254:43879)
0812 11:26:57.600792 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:57.600798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:57.601742 (+ 944us) server_negotiation.cc:407] Connection header received
0812 11:26:57.603632 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:57.603636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:57.603709 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:57.603830 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:57.605350 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.606158 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.607782 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.608244 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.608344 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:57.611899 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:57.611904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:57.611907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:57.611949 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:57.615667 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.616417 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.616421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.616423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.616545 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.616989 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.616998 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.617000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.617257 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:57.617383 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:57.618034 (+ 651us) server_negotiation.cc:299] Negotiation successful
0812 11:26:57.618190 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":277,"threads_started":1}
I20260812 11:26:57.619386 2365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:57.600708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:43879 (local address 127.12.110.195:60269)
0812 11:26:57.601570 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:57.601616 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:57.603362 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:57.604076 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:57.604084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:57.604515 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:26:57.605178 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:57.605195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.606330 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.606334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:57.607629 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:57.607642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.608460 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.608466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:57.608668 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:57.609308 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:26:57.609329 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:26:57.611668 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:26:57.615799 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:57.615806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:57.615818 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:57.616307 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:57.616641 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:57.616644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:57.616647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:57.616865 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:57.617367 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:26:57.617372 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:26:57.617664 (+ 292us) client_negotiation.cc:769] Sending connection context
0812 11:26:57.617885 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 11:26:57.618244 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":368,"threads_started":1}
I20260812 11:26:57.620680 2357 heartbeater.cc:347] Connected to a master server at 127.12.110.254:43879
I20260812 11:26:57.620978 2357 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:57.621660 2357 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
I20260812 11:26:57.623118 1454 ts_manager.cc:194] Registered new tserver with Master: d69b920832674b68beecdb748b7991ca (127.12.110.195:43041)
I20260812 11:26:57.623921 1454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:60269
I20260812 11:26:57.627775 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:26:57.629496 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 1592
Aug 12 11:26:57 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:26:57.653662 2368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:57.630447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40712 (local address 127.12.110.193:45097)
0812 11:26:57.631444 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:57.631453 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:57.631482 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:26:57.631549 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:57.631554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:57.631732 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:57.631852 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:57.633779 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.634893 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.636574 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.637739 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.637988 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:57.645251 (+ 7263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:57.645266 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:57.645281 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:57.645325 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:57.649403 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.650872 (+ 1469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.650892 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.650895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.651129 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.651704 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.651718 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.651721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.652183 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:57.652590 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:57.652947 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 11:26:57.653267 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":523,"threads_started":1}
I20260812 11:26:57.654974 1841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:57.738409 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 1592
I20260812 11:26:57.759287 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 1856
Aug 12 11:26:57 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:26:57.781847 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:57.764923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54276 (local address 127.12.110.194:43913)
0812 11:26:57.765512 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:57.765519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:57.765548 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:26:57.765617 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:57.765623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:57.765858 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:57.766009 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:57.767483 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.768784 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.770204 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.771018 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.771125 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:57.775607 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:57.775620 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:57.775635 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:57.775672 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:57.778780 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.779520 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.779528 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.779530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.779683 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.780175 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.780182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.780185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.780682 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:57.780935 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:57.781207 (+ 272us) server_negotiation.cc:299] Negotiation successful
0812 11:26:57.781428 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":311,"threads_started":1}
I20260812 11:26:57.783257 2101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:57.869125 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 1856
I20260812 11:26:57.898154 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 2122
Aug 12 11:26:57 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:26:57.918962 2374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:57.899294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.12.110.195:43041)
0812 11:26:57.900075 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:57.900081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:57.900111 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:26:57.900173 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:57.900179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:57.900382 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:57.900512 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:57.901901 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.903154 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.905111 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:57.905653 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:57.905823 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:57.911249 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:57.911260 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:57.911274 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:57.911314 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:57.914384 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.915421 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.915441 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.915444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.915719 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:57.916302 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:57.916307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:57.916310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:57.916728 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:57.917019 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:57.917833 (+ 814us) server_negotiation.cc:299] Negotiation successful
0812 11:26:57.918070 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":436,"threads_started":1}
I20260812 11:26:57.920280 2350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:58.003638 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 2122
I20260812 11:26:58.023940 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 1395
I20260812 11:26:58.025576 1511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:26:58.133173 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 1395
I20260812 11:26:58.165706 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43879
--webserver_interface=127.12.110.254
--webserver_port=34025
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:43879
--ranger_config_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:26:58.534227 2381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:58.534626 2381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:58.534801 2381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:58.542476 2381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:26:58.542716 2381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:58.542816 2381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:58.542928 2381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:26:58.543035 2381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:26:58.553519 2381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:43879
--ranger_config_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=34025
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:58.554924 2381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:58.556439 2381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:58.570107 2402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:58.570226 2393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:58.570637 2391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:58.571118 2381 server_base.cc:1034] running on GCE node
I20260812 11:26:58.571988 2381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:58.573668 2381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:58.574957 2381 hybrid_clock.cc:648] HybridClock initialized: now 1786534018574896 us; error 48 us; skew 500 ppm
I20260812 11:26:58.575368 2381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:58 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534018, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:58.578858 2381 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:26:58.580482 2381 webserver.cc:460] Webserver started at http://127.12.110.254:34025/ using document root <none> and password file <none>
I20260812 11:26:58.581197 2381 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:58.581368 2381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:26:58 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534018, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:58.668952 2381 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:26:58.672300 2415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:58.674127 2381 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260812 11:26:58.674381 2381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "40fffcc3251a40fa9c96671c588a1d45"
format_stamp: "Formatted at 2026-08-12 11:26:54 on dist-test-slave-h6n0"
server_key: "4021a82e0da78699ee88ba793b4a0cd1"
server_key_iv: "5d2b0f90a9f1d9a3fc3d4e1d50033a3f"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:58.675485 2381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:58.702234 2381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:58.707679 2381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:58.708036 2381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:58.730342 2381 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:43879
I20260812 11:26:58.730367 2514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:43879 every 8 connection(s)
I20260812 11:26:58.731989 2381 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:26:58.738816 2515 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:26:58.739528 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 2381
I20260812 11:26:58.740686 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:45097
--local_ip_for_outbound_sockets=127.12.110.193
--tserver_master_addrs=127.12.110.254:43879
--webserver_port=44675
--webserver_interface=127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:26:58.743947 2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Bootstrap starting.
I20260812 11:26:58.767407 2515 log.cc:824] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Log is configured to *not* fsync() on all Append() calls
I20260812 11:26:58.776805 2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:26:58.777463 2515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Bootstrap complete.
I20260812 11:26:58.787397 2515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:58.788467 2515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40fffcc3251a40fa9c96671c588a1d45, State: Initialized, Role: FOLLOWER
I20260812 11:26:58.789095 2515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:58.789340 2515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:26:58.789474 2515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:26:58.789651 2515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:26:58.793597 2515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:58.794078 2515 leader_election.cc:302] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40fffcc3251a40fa9c96671c588a1d45; no voters:
I20260812 11:26:58.794736 2515 leader_election.cc:288] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:26:58.794873 2520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:26:58.796248 2520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [term 2 LEADER]: Becoming Leader. State: Replica: 40fffcc3251a40fa9c96671c588a1d45, State: Running, Role: LEADER
I20260812 11:26:58.796842 2520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } }
I20260812 11:26:58.797137 2515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:26:58.800395 2523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40fffcc3251a40fa9c96671c588a1d45. Latest consensus state: current_term: 2 leader_uuid: "40fffcc3251a40fa9c96671c588a1d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } } }
I20260812 11:26:58.800916 2523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: This master's current role is: LEADER
I20260812 11:26:58.801635 2521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "40fffcc3251a40fa9c96671c588a1d45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40fffcc3251a40fa9c96671c588a1d45" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43879 } } }
I20260812 11:26:58.802034 2521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45 [sys.catalog]: This master's current role is: LEADER
I20260812 11:26:58.804567 2529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:26:58.811790 2529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:26:58.813783 2529 catalog_manager.cc:1259] Loaded cluster ID: 1939c77778474ed29926f7c39663b0de
I20260812 11:26:58.813984 2529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:26:58.818559 2529 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:26:58.820096 2529 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Loaded TSK: 0
I20260812 11:26:58.822342 2529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:26:58.828667 2515 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260812 11:26:59.173648 2518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:59.174187 2518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:59.174410 2518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:59.188719 2518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:59.189003 2518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:59.189308 2518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:26:59.207211 2518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:45097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=44675
--tserver_master_addrs=127.12.110.254:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:59.209408 2518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:59.211311 2518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:59.247167 2575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:59.247653 2577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:59.258033 2581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:59.260185 2518 server_base.cc:1034] running on GCE node
I20260812 11:26:59.261104 2518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:59.262677 2518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:59.264025 2518 hybrid_clock.cc:648] HybridClock initialized: now 1786534019263946 us; error 60 us; skew 500 ppm
I20260812 11:26:59.264602 2518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:59 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:59.269476 2518 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:26:59.281996 2518 webserver.cc:460] Webserver started at http://127.12.110.193:44675/ using document root <none> and password file <none>
I20260812 11:26:59.283102 2518 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:59.283291 2518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:26:59 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:59.308130 2518 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:26:59.311054 2593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:59.313051 2518 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260812 11:26:59.313271 2518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "b8df244fc61741e59f885648276269e7"
format_stamp: "Formatted at 2026-08-12 11:26:55 on dist-test-slave-h6n0"
server_key: "144810609ac666786bb5591bd754a770"
server_key_iv: "0fcb31a8a840a69a5861542835fe3e11"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:59.314293 2518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:26:59.335883 2518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:26:59.341480 2518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:26:59.341863 2518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:26:59.345875 2518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:26:59.348270 2518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:26:59.348423 2518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260812 11:26:59.348577 2518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:26:59.348708 2518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:59.467564 2518 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:45097
I20260812 11:26:59.467614 2792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:45097 every 8 connection(s)
I20260812 11:26:59.469231 2518 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:26:59.472626 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 2518
I20260812 11:26:59.473623 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:43913
--local_ip_for_outbound_sockets=127.12.110.194
--tserver_master_addrs=127.12.110.254:43879
--webserver_port=40591
--webserver_interface=127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:26:59 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:26:59.508790 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:59.473346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:43879 (local address 127.12.110.193:34389)
0812 11:26:59.474863 (+ 1517us) negotiation.cc:107] Waiting for socket to connect
0812 11:26:59.474925 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0812 11:26:59.477741 (+ 2816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:26:59.482463 (+ 4722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:26:59.482474 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:26:59.483502 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:26:59.484220 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:59.484244 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:59.486761 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:59.486766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:59.488786 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:26:59.488802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:26:59.490654 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:26:59.490664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:26:59.490965 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:59.491810 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:26:59.491861 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:26:59.498081 (+ 6220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:26:59.503336 (+ 5255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:59.503347 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:59.503363 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:59.504031 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:59.504593 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:26:59.504601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:26:59.504607 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:26:59.504875 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:26:59.506018 (+ 1143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:26:59.506039 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:26:59.506466 (+ 427us) client_negotiation.cc:769] Sending connection context
0812 11:26:59.506804 (+ 338us) client_negotiation.cc:240] Negotiation successful
0812 11:26:59.507375 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1195,"thread_start_us":1506,"threads_started":1}
I20260812 11:26:59.508929 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:26:59.474731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:34389 (local address 127.12.110.254:43879)
0812 11:26:59.480210 (+ 5479us) server_negotiation.cc:206] Beginning negotiation
0812 11:26:59.480229 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0812 11:26:59.480282 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:26:59.480748 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:26:59.480768 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:26:59.481255 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:26:59.481634 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:26:59.484502 (+ 2868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:59.486450 (+ 1948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:59.488964 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:26:59.489958 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:26:59.490136 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:26:59.498417 (+ 8281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:26:59.498429 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:26:59.498450 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:26:59.498520 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:26:59.503127 (+ 4607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:59.504210 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:59.504221 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:59.504231 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:59.504429 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:26:59.505014 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:26:59.505021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:26:59.505027 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:26:59.505822 (+ 795us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:26:59.506029 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0812 11:26:59.507035 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0812 11:26:59.507574 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5199,"thread_start_us":4957,"threads_started":1}
I20260812 11:26:59.514079 2793 heartbeater.cc:347] Connected to a master server at 127.12.110.254:43879
I20260812 11:26:59.514452 2793 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:59.515611 2793 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
W20260812 11:26:59.519431 2793 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:43879 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 11:26:59.519676 2793 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:59.519982 2793 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
I20260812 11:26:59.521364 2793 heartbeater.cc:464] Registering TS with master...
I20260812 11:26:59.521689 2793 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
W20260812 11:26:59.522775 2793 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:26:59.903594 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:26:59.904227 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:26:59.904502 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:26:59.912431 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:26:59.912607 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:26:59.912794 2807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:26:59.924113 2807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:43913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=40591
--tserver_master_addrs=127.12.110.254:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:26:59.925595 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:26:59.926980 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:26:59.940922 2835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:59.940966 2834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:26:59.940811 2838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:26:59.942837 2807 server_base.cc:1034] running on GCE node
I20260812 11:26:59.943531 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:26:59.944553 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:26:59.945760 2807 hybrid_clock.cc:648] HybridClock initialized: now 1786534019945711 us; error 43 us; skew 500 ppm
I20260812 11:26:59.946089 2807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:26:59 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:26:59.949604 2807 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:26:59.951727 2807 webserver.cc:460] Webserver started at http://127.12.110.194:40591/ using document root <none> and password file <none>
I20260812 11:26:59.952414 2807 fs_manager.cc:359] Metadata directory not provided
I20260812 11:26:59.952577 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:26:59 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:26:59.975893 2807 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:26:59.978935 2854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:26:59.980471 2807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:26:59.980644 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "dd03439b6ded4fb99b5a0df47e19a0bc"
format_stamp: "Formatted at 2026-08-12 11:26:56 on dist-test-slave-h6n0"
server_key: "fe460e4a577d3bf70efdec463b53aa8a"
server_key_iv: "6d78bd5a2cb007d4654928a1f9ac97fc"
server_key_version: "kuduclusterkey@0"
I20260812 11:26:59.981680 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:00.009610 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:00.014928 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:00.015275 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:00.016580 2807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:00.018332 2807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:00.018491 2807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:00.018651 2807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:00.018765 2807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:00.061230 2807 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:43913
I20260812 11:27:00.061316 3052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:43913 every 8 connection(s)
I20260812 11:27:00.063190 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:27:00.070279 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 2807
I20260812 11:27:00.071357 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.12.110.212:39943/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:43041
--local_ip_for_outbound_sockets=127.12.110.195
--tserver_master_addrs=127.12.110.254:43879
--webserver_port=35695
--webserver_interface=127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:27:00 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534019, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:00.132275 3058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.066723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:43879 (local address 127.12.110.194:53353)
0812 11:27:00.101011 (+ 34288us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:00.101083 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:00.104554 (+ 3471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:00.113429 (+ 8875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:00.113443 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:00.114121 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:00.115095 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.115129 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.116889 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.116897 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.119247 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.119261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.120070 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.120080 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.120323 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.121035 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:00.121058 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:00.123347 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:00.127558 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:00.127568 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:00.127585 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:00.128250 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:00.128707 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:00.128713 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:00.128717 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:00.129007 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:00.129776 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:00.129784 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:00.130208 (+ 424us) client_negotiation.cc:769] Sending connection context
0812 11:27:00.130512 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 11:27:00.131096 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31392,"thread_start_us":31744,"threads_started":1}
I20260812 11:27:00.133388 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.065989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:53353 (local address 127.12.110.254:43879)
0812 11:27:00.112726 (+ 46737us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:00.112734 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:00.112758 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:27:00.112833 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:00.112843 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:00.112939 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:00.113080 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:00.115343 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.116652 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.119393 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.119819 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.119961 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.123612 (+ 3651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:00.123621 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:00.123629 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:00.123675 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:00.127367 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.128402 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.128407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.128411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.128571 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.129151 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.129157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.129160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.129535 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:00.129698 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:00.132992 (+ 3294us) server_negotiation.cc:299] Negotiation successful
0812 11:27:00.133159 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46577,"thread_start_us":46428,"threads_started":1}
I20260812 11:27:00.134884 3054 heartbeater.cc:347] Connected to a master server at 127.12.110.254:43879
I20260812 11:27:00.135210 3054 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:00.136092 3054 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
I20260812 11:27:00.138129 2447 ts_manager.cc:194] Registered new tserver with Master: dd03439b6ded4fb99b5a0df47e19a0bc (127.12.110.194:43913)
I20260812 11:27:00.140337 2447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:53353
Aug 12 11:27:00 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786534020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:00.523021 2793 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:00.523881 2793 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
I20260812 11:27:00.525291 2447 ts_manager.cc:194] Registered new tserver with Master: b8df244fc61741e59f885648276269e7 (127.12.110.193:45097)
I20260812 11:27:00.526463 2447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:34389
W20260812 11:27:00.549592 3063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:00.550060 3063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:00.550267 3063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:00.561745 3063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:00.561879 3063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:00.562057 3063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:27:00.579118 3063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.12.110.212:39943/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:43041
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=35695
--tserver_master_addrs=127.12.110.254:43879
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:00.581024 3063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:00.582679 3063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:00.603116 3075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:00.603323 3076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:00.613139 3083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:00.613883 3063 server_base.cc:1034] running on GCE node
I20260812 11:27:00.614812 3063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:00.616127 3063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:00.617408 3063 hybrid_clock.cc:648] HybridClock initialized: now 1786534020617345 us; error 42 us; skew 500 ppm
I20260812 11:27:00.617830 3063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:00 dist-test-slave-h6n0 krb5kdc[588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:00.622217 3063 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:27:00.628021 3063 webserver.cc:460] Webserver started at http://127.12.110.195:35695/ using document root <none> and password file <none>
I20260812 11:27:00.629020 3063 fs_manager.cc:359] Metadata directory not provided
Aug 12 11:27:00 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:27:00.629182 3063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 12 11:27:00 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:27:00.660131 3063 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.007s sys 0.004s
I20260812 11:27:00.664055 3094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:00.669931 3063 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.001s
I20260812 11:27:00.671067 3063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "d69b920832674b68beecdb748b7991ca"
format_stamp: "Formatted at 2026-08-12 11:26:57 on dist-test-slave-h6n0"
server_key: "f058135e88b5e77a200f18c365de5aa8"
server_key_iv: "afcd9bde752eed7b5973729c22ac5fa5"
server_key_version: "kuduclusterkey@0"
I20260812 11:27:00.685606 3063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:00.726047 3063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:00.731047 3063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:00.731539 3063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:00.742424 3063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:00.744334 3063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:00.744450 3063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:00.744571 3063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:00.744653 3063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:00.796353 3063 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:43041
I20260812 11:27:00.796422 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:43041 every 8 connection(s)
I20260812 11:27:00.798486 3063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:27:00.800958 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 3063
I20260812 11:27:00.848343 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.801984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.12.110.254:43879)
0812 11:27:00.823987 (+ 22003us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:00.823993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:00.824015 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:27:00.824085 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:00.824092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:00.824190 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:00.824306 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:00.826437 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.827678 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.832614 (+ 4936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.833377 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.833513 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.839507 (+ 5994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:00.839515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:00.839520 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:00.839579 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:00.843644 (+ 4065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.844597 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.844603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.844606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.844773 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.845715 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.845721 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.845725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.846100 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:00.846379 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:00.847683 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0812 11:27:00.847921 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21879,"thread_start_us":43152,"threads_started":1}
Aug 12 11:27:00 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:00.867007 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.803791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:42171 (local address 127.12.110.254:43879)
0812 11:27:00.845569 (+ 41778us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:00.845576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:00.845611 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:27:00.845722 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:00.845729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:00.845865 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:00.848762 (+ 2897us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:00.849256 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.850875 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.853345 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.854025 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.854161 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.858610 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:00.858619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:00.858624 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:00.858665 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:00.862197 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.863544 (+ 1347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.863551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.863554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.863692 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.864619 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.864625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.864629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.865055 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:00.865223 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:00.866234 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0812 11:27:00.866485 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41581,"thread_start_us":19875,"threads_started":1}
I20260812 11:27:00.868165 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.803135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:43879 (local address 127.12.110.195:42171)
0812 11:27:00.822763 (+ 19628us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:00.822832 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:00.825367 (+ 2535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:00.846832 (+ 21465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:00.846850 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:00.847643 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:00.848984 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.849016 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.851059 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.851064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.853166 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.853201 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.854278 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.854285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.854575 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.855383 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:00.855411 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:00.858187 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:00.862482 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:00.862491 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:00.862509 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:00.863393 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:00.863816 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:00.863821 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:00.863824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:00.864451 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:00.865249 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:00.865257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:00.865729 (+ 472us) client_negotiation.cc:769] Sending connection context
0812 11:27:00.866072 (+ 343us) client_negotiation.cc:240] Negotiation successful
0812 11:27:00.866680 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19306,"thread_start_us":19057,"threads_started":1}
I20260812 11:27:00.869864 3283 heartbeater.cc:347] Connected to a master server at 127.12.110.254:43879
I20260812 11:27:00.870229 3283 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:00.871177 3283 heartbeater.cc:511] Master 127.12.110.254:43879 requested a full tablet report, sending...
I20260812 11:27:00.873059 2447 ts_manager.cc:194] Registered new tserver with Master: d69b920832674b68beecdb748b7991ca (127.12.110.195:43041)
I20260812 11:27:00.874303 2447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:42171
I20260812 11:27:00.876987 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:27:00.900548 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.881072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.12.110.254:43879)
0812 11:27:00.881249 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:00.881255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:00.883096 (+ 1841us) server_negotiation.cc:407] Connection header received
0812 11:27:00.883250 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:00.883256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:00.883358 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:00.883478 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:00.885455 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.886701 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.888658 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.889357 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.889491 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.893217 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:00.893224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:00.893230 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:00.893273 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:00.896704 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.897515 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.897520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.897524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.897690 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:00.898090 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:00.898095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:00.898098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:00.898458 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:00.898784 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:00.899941 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0812 11:27:00.900198 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:27:00.907598 2447 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:27:00.918848 2447 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:27:00.962441 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.943429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:45097 (local address 127.0.0.1:33502)
0812 11:27:00.951988 (+ 8559us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:00.952005 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:00.952110 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:00.953616 (+ 1506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:00.953623 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:00.953673 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:00.954149 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.954166 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.956598 (+ 2432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.956606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.959478 (+ 2872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.959499 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.961552 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.961559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.961748 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.961802 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 11:27:00.961925 (+ 123us) client_negotiation.cc:240] Negotiation successful
0812 11:27:00.962107 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8418,"thread_start_us":7295,"threads_started":1}
I20260812 11:27:00.963449 3312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.944025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.12.110.193:45097)
0812 11:27:00.952646 (+ 8621us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:00.952656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:00.952686 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:27:00.952785 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:00.952797 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:00.953083 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:00.953289 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:00.954384 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.956404 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.959674 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.961053 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.961235 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.961398 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:00.962965 (+ 1567us) server_negotiation.cc:299] Negotiation successful
0812 11:27:00.963142 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8422,"thread_start_us":14504,"threads_started":1}
I20260812 11:27:00.967509 2679 tablet_service.cc:1467] Processing CreateTablet for tablet e4ac382643a24a5c93cf186fc50bd0db (DEFAULT_TABLE table=test-table [id=a31987a895744ec98aab4eb3a212caf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:00.969419 2679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ac382643a24a5c93cf186fc50bd0db. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 12 11:27:00 dist-test-slave-h6n0 krb5kdc[588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534020, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for HTTP/127.12.110.212@KRBTEST.COM
I20260812 11:27:00.998143 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.941953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:43913 (local address 127.0.0.1:49950)
0812 11:27:00.951128 (+ 9175us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:00.951181 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:00.951366 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:00.959683 (+ 8317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:00.959690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:00.959726 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:00.960185 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.960199 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.975329 (+ 15130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.975340 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.978059 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.978078 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.997293 (+ 19215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.997300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.997474 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.997523 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 11:27:00.997622 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:27:00.997779 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8913,"thread_start_us":9749,"threads_started":1}
I20260812 11:27:00.999377 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.941996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.12.110.194:43913)
0812 11:27:00.958186 (+ 16190us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:00.958194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:00.958220 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:27:00.958308 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:00.958314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:00.958564 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:00.958754 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:00.966065 (+ 7311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.975100 (+ 9035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.978290 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.979857 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.997151 (+ 17294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:00.997391 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:00.998841 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0812 11:27:00.999049 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16020,"thread_start_us":16946,"threads_started":1}
I20260812 11:27:01.003791 2931 tablet_service.cc:1467] Processing CreateTablet for tablet e4ac382643a24a5c93cf186fc50bd0db (DEFAULT_TABLE table=test-table [id=a31987a895744ec98aab4eb3a212caf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:01.004431 3309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.952725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:43041 (local address 127.0.0.1:46138)
0812 11:27:00.962712 (+ 9987us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:00.962729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:00.962817 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:00.991972 (+ 29155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:00.991977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:00.992012 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:00.992230 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.992242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.994629 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.994636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:00.996566 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:00.996583 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.003677 (+ 7094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.003685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.003888 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.003927 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:27:01.004000 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:27:01.004149 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9907,"thread_start_us":25298,"threads_started":1}
I20260812 11:27:01.005621 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:00.953472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46138 (local address 127.12.110.195:43041)
0812 11:27:00.991091 (+ 37619us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:00.991102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:00.991143 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:27:00.991272 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:00.991280 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:00.991554 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:00.992415 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:00.992763 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:00.994424 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:00.996814 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.003466 (+ 6652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.004842 (+ 1376us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.005023 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.005096 (+ 73us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.005255 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37408,"thread_start_us":37715,"threads_started":1}
I20260812 11:27:01.005649 2931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ac382643a24a5c93cf186fc50bd0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:01.007455 3319 tablet_bootstrap.cc:492] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Bootstrap starting.
I20260812 11:27:01.009339 3171 tablet_service.cc:1467] Processing CreateTablet for tablet e4ac382643a24a5c93cf186fc50bd0db (DEFAULT_TABLE table=test-table [id=a31987a895744ec98aab4eb3a212caf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:01.010839 3171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4ac382643a24a5c93cf186fc50bd0db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:01.012861 3319 tablet_bootstrap.cc:654] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:01.014470 3319 log.cc:824] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:01.041965 3319 tablet_bootstrap.cc:492] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: No bootstrap required, opened a new log
I20260812 11:27:01.042431 3319 ts_tablet_manager.cc:1397] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.008s
I20260812 11:27:01.053264 3319 raft_consensus.cc:348] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.053933 3319 raft_consensus.cc:374] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:01.054111 3319 raft_consensus.cc:729] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8df244fc61741e59f885648276269e7, State: Initialized, Role: FOLLOWER
I20260812 11:27:01.054983 3319 consensus_queue.cc:260] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.071409 3324 tablet_bootstrap.cc:492] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Bootstrap starting.
I20260812 11:27:01.074048 2793 heartbeater.cc:503] Master 127.12.110.254:43879 was elected leader, sending a full tablet report...
I20260812 11:27:01.075299 3325 tablet_bootstrap.cc:492] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Bootstrap starting.
I20260812 11:27:01.079082 3328 raft_consensus.cc:482] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:27:01.079622 3328 raft_consensus.cc:504] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.097218 3325 tablet_bootstrap.cc:654] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:01.098033 3324 tablet_bootstrap.cc:654] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:01.100916 3325 log.cc:824] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:01.102288 3324 log.cc:824] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:01.107113 3328 leader_election.cc:288] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd03439b6ded4fb99b5a0df47e19a0bc (127.12.110.194:43913), d69b920832674b68beecdb748b7991ca (127.12.110.195:43041)
I20260812 11:27:01.120612 3319 ts_tablet_manager.cc:1428] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Time spent starting tablet: real 0.078s user 0.017s sys 0.006s
I20260812 11:27:01.122807 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.082260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:48229 (local address 127.12.110.194:43913)
0812 11:27:01.082876 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.082882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.105287 (+ 22405us) server_negotiation.cc:407] Connection header received
0812 11:27:01.105992 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.105998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.106070 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.106126 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:01.108613 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.109660 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.116921 (+ 7261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.118344 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.118481 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.118594 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.122239 (+ 3645us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.122399 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260812 11:27:01.123925 3331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.081848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:43913 (local address 127.12.110.193:48229)
0812 11:27:01.105174 (+ 23326us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:01.105195 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:01.105623 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:01.106351 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:01.106357 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:01.106381 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:01.106762 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.106775 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.109876 (+ 3101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.109884 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.112104 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.112124 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.118894 (+ 6770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.118900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.119076 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.119111 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:27:01.123262 (+ 4151us) client_negotiation.cc:240] Negotiation successful
0812 11:27:01.123419 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23184,"thread_start_us":25983,"threads_started":1}
I20260812 11:27:01.126967 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.094616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:48257 (local address 127.12.110.195:43041)
0812 11:27:01.094845 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.094851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.107816 (+ 12965us) server_negotiation.cc:407] Connection header received
0812 11:27:01.112356 (+ 4540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.112365 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.112472 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.112556 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:01.113304 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.114410 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.124277 (+ 9867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.125615 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.125793 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.125901 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.126581 (+ 680us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.126722 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260812 11:27:01.129113 2966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ac382643a24a5c93cf186fc50bd0db" candidate_uuid: "b8df244fc61741e59f885648276269e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" is_pre_election: true
W20260812 11:27:01.134815 2598 leader_election.cc:341] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd03439b6ded4fb99b5a0df47e19a0bc (127.12.110.194:43913): Illegal state: must be running to vote when last-logged opid is not known
I20260812 11:27:01.127950 3329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.094158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:43041 (local address 127.12.110.193:48257)
0812 11:27:01.107712 (+ 13554us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:01.107731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:01.108425 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:01.112755 (+ 4330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:01.112760 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:01.112786 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:01.113104 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.113115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.114631 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.114636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.116717 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.116734 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.126134 (+ 9400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.126141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.126346 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.126385 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:27:01.127399 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0812 11:27:01.127580 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13438,"thread_start_us":11139,"threads_started":1}
I20260812 11:27:01.136636 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ac382643a24a5c93cf186fc50bd0db" candidate_uuid: "b8df244fc61741e59f885648276269e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d69b920832674b68beecdb748b7991ca" is_pre_election: true
I20260812 11:27:01.141163 3325 tablet_bootstrap.cc:492] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: No bootstrap required, opened a new log
I20260812 11:27:01.141492 3325 ts_tablet_manager.cc:1397] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.011s
W20260812 11:27:01.141686 2600 leader_election.cc:341] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d69b920832674b68beecdb748b7991ca (127.12.110.195:43041): Illegal state: must be running to vote when last-logged opid is not known
I20260812 11:27:01.141884 2600 leader_election.cc:302] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b8df244fc61741e59f885648276269e7; no voters: d69b920832674b68beecdb748b7991ca, dd03439b6ded4fb99b5a0df47e19a0bc
I20260812 11:27:01.142724 3327 raft_consensus.cc:2726] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260812 11:27:01.144805 3054 heartbeater.cc:503] Master 127.12.110.254:43879 was elected leader, sending a full tablet report...
I20260812 11:27:01.150270 3324 tablet_bootstrap.cc:492] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: No bootstrap required, opened a new log
I20260812 11:27:01.150547 3324 ts_tablet_manager.cc:1397] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.007s
I20260812 11:27:01.165488 3324 raft_consensus.cc:348] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.166035 3324 raft_consensus.cc:374] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:01.166167 3324 raft_consensus.cc:729] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd03439b6ded4fb99b5a0df47e19a0bc, State: Initialized, Role: FOLLOWER
I20260812 11:27:01.166889 3324 consensus_queue.cc:260] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.169201 3324 ts_tablet_manager.cc:1428] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Time spent starting tablet: real 0.019s user 0.004s sys 0.006s
I20260812 11:27:01.172135 3325 raft_consensus.cc:348] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.172564 3325 raft_consensus.cc:374] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:01.172663 3325 raft_consensus.cc:729] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d69b920832674b68beecdb748b7991ca, State: Initialized, Role: FOLLOWER
I20260812 11:27:01.173296 3325 consensus_queue.cc:260] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.174353 3283 heartbeater.cc:503] Master 127.12.110.254:43879 was elected leader, sending a full tablet report...
I20260812 11:27:01.175734 3325 ts_tablet_manager.cc:1428] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Time spent starting tablet: real 0.034s user 0.010s sys 0.000s
W20260812 11:27:01.232075 2795 tablet.cc:2369] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:27:01.321509 3056 tablet.cc:2369] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:27:01.339835 3285 tablet.cc:2369] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:27:01.497268 3336 raft_consensus.cc:482] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:27:01.497673 3336 raft_consensus.cc:504] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.499480 3336 leader_election.cc:288] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b8df244fc61741e59f885648276269e7 (127.12.110.193:45097), d69b920832674b68beecdb748b7991ca (127.12.110.195:43041)
I20260812 11:27:01.507109 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.499996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:46179 (local address 127.12.110.195:43041)
0812 11:27:01.500252 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.500262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.501317 (+ 1055us) server_negotiation.cc:407] Connection header received
0812 11:27:01.501501 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.501505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.501570 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.501667 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:01.502271 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.503064 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.504946 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.505990 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.506113 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.506165 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.506630 (+ 465us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.506793 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260812 11:27:01.507143 3356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.500054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:43041 (local address 127.12.110.194:46179)
0812 11:27:01.501220 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:01.501237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:01.501358 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:01.501689 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:01.501693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:01.501718 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:01.502111 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.502121 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.503198 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.503203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.504757 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.504773 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.506200 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.506206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.506378 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.506440 (+ 62us) client_negotiation.cc:769] Sending connection context
0812 11:27:01.506578 (+ 138us) client_negotiation.cc:240] Negotiation successful
0812 11:27:01.506716 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1015,"mutex_wait_us":3,"thread_start_us":722,"threads_started":1}
I20260812 11:27:01.507864 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.499807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:47125 (local address 127.12.110.193:45097)
0812 11:27:01.501519 (+ 1712us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.501524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.501541 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:27:01.501606 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.501609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.501663 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.501785 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:01.502476 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.503256 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.505688 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.506746 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.506905 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.506975 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.507387 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.507557 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1590,"thread_start_us":1408,"threads_started":1}
I20260812 11:27:01.507817 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ac382643a24a5c93cf186fc50bd0db" candidate_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d69b920832674b68beecdb748b7991ca" is_pre_election: true
I20260812 11:27:01.508092 3354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.499694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:45097 (local address 127.12.110.194:47125)
0812 11:27:01.500823 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:01.500847 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:01.501106 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:01.501807 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:01.501811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:01.501855 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:01.502316 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.502327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.503395 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.503401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.505528 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:01.505544 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.506971 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.506978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:01.507174 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.507207 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:27:01.507308 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 11:27:01.507461 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":21,"thread_start_us":778,"threads_started":1}
I20260812 11:27:01.508232 3204 raft_consensus.cc:2454] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd03439b6ded4fb99b5a0df47e19a0bc in term 0.
I20260812 11:27:01.508772 2861 leader_election.cc:302] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d69b920832674b68beecdb748b7991ca, dd03439b6ded4fb99b5a0df47e19a0bc; no voters:
I20260812 11:27:01.508989 2711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ac382643a24a5c93cf186fc50bd0db" candidate_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8df244fc61741e59f885648276269e7" is_pre_election: true
I20260812 11:27:01.509136 3336 raft_consensus.cc:2781] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:27:01.509274 3336 raft_consensus.cc:482] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:27:01.509385 3336 raft_consensus.cc:3037] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:01.509388 2711 raft_consensus.cc:2454] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd03439b6ded4fb99b5a0df47e19a0bc in term 0.
I20260812 11:27:01.511904 3336 raft_consensus.cc:504] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.512562 3336 leader_election.cc:288] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [CANDIDATE]: Term 1 election: Requested vote from peers b8df244fc61741e59f885648276269e7 (127.12.110.193:45097), d69b920832674b68beecdb748b7991ca (127.12.110.195:43041)
I20260812 11:27:01.512924 2711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ac382643a24a5c93cf186fc50bd0db" candidate_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8df244fc61741e59f885648276269e7"
I20260812 11:27:01.513120 2711 raft_consensus.cc:3037] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:01.513085 3204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4ac382643a24a5c93cf186fc50bd0db" candidate_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d69b920832674b68beecdb748b7991ca"
I20260812 11:27:01.513343 3204 raft_consensus.cc:3037] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:01.515558 3204 raft_consensus.cc:2454] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd03439b6ded4fb99b5a0df47e19a0bc in term 1.
I20260812 11:27:01.515568 2711 raft_consensus.cc:2454] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd03439b6ded4fb99b5a0df47e19a0bc in term 1.
I20260812 11:27:01.515937 2861 leader_election.cc:302] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d69b920832674b68beecdb748b7991ca, dd03439b6ded4fb99b5a0df47e19a0bc; no voters:
I20260812 11:27:01.516243 3336 raft_consensus.cc:2781] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:27:01.516708 3336 raft_consensus.cc:686] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 1 LEADER]: Becoming Leader. State: Replica: dd03439b6ded4fb99b5a0df47e19a0bc, State: Running, Role: LEADER
I20260812 11:27:01.517213 3336 consensus_queue.cc:237] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } }
I20260812 11:27:01.522071 2447 catalog_manager.cc:5495] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to dd03439b6ded4fb99b5a0df47e19a0bc (127.12.110.194). New cstate: current_term: 1 leader_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd03439b6ded4fb99b5a0df47e19a0bc" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 43913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 } health_report { overall_health: UNKNOWN } } }
I20260812 11:27:01.537737 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.532293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.12.110.194:43913)
0812 11:27:01.532540 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.532547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.532715 (+ 168us) server_negotiation.cc:407] Connection header received
0812 11:27:01.532885 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.532889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.532963 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.533120 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:27:01.533446 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.534238 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.535638 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.536047 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.536143 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.536629 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:27:01.537113 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:27:01.537218 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.537346 (+ 128us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.537409 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:27:01.545559 3204 raft_consensus.cc:1261] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 1 FOLLOWER]: Refusing update from remote peer dd03439b6ded4fb99b5a0df47e19a0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:01.545634 2711 raft_consensus.cc:1261] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 1 FOLLOWER]: Refusing update from remote peer dd03439b6ded4fb99b5a0df47e19a0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:01.546868 3357 consensus_queue.cc:1035] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b8df244fc61741e59f885648276269e7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 45097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:01.547329 3336 consensus_queue.cc:1035] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d69b920832674b68beecdb748b7991ca" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:01.558033 3366 mvcc.cc:205] Tried to move back new op lower bound from 7317643352241590272 to 7317643352138326016. Current Snapshot: MvccSnapshot[applied={T|T < 7317643352241590272}]
I20260812 11:27:01.558357 3369 mvcc.cc:205] Tried to move back new op lower bound from 7317643352241590272 to 7317643352138326016. Current Snapshot: MvccSnapshot[applied={T|T < 7317643352241590272}]
I20260812 11:27:01.559844 3370 mvcc.cc:205] Tried to move back new op lower bound from 7317643352241590272 to 7317643352138326016. Current Snapshot: MvccSnapshot[applied={T|T < 7317643352241590272}]
I20260812 11:27:01.573145 2443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:01.573439 2443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:01.576944 2443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40fffcc3251a40fa9c96671c588a1d45: Sending DeleteTablet for 3 replicas of tablet e4ac382643a24a5c93cf186fc50bd0db
I20260812 11:27:01.578361 2931 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ac382643a24a5c93cf186fc50bd0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:01 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:49950
I20260812 11:27:01.578433 2679 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ac382643a24a5c93cf186fc50bd0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:01 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:33502
I20260812 11:27:01.579185 3171 tablet_service.cc:1514] Processing DeleteTablet for tablet e4ac382643a24a5c93cf186fc50bd0db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:01 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:46138
I20260812 11:27:01.579419 3383 tablet_replica.cc:330] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: stopping tablet replica
I20260812 11:27:01.579828 3383 raft_consensus.cc:2229] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:27:01.580572 3383 raft_consensus.cc:2258] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:01.581331 3385 tablet_replica.cc:330] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: stopping tablet replica
I20260812 11:27:01.581816 3385 raft_consensus.cc:2229] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:01.582343 3385 raft_consensus.cc:2258] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:01.582933 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 2518
I20260812 11:27:01.584764 3385 ts_tablet_manager.cc:1905] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:01.591233 3386 tablet_replica.cc:330] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: stopping tablet replica
I20260812 11:27:01.592082 3385 ts_tablet_manager.cc:1918] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:01.592337 3385 log.cc:1196] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/e4ac382643a24a5c93cf186fc50bd0db
I20260812 11:27:01.592945 3383 ts_tablet_manager.cc:1905] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:01.593138 3385 ts_tablet_manager.cc:1939] T e4ac382643a24a5c93cf186fc50bd0db P b8df244fc61741e59f885648276269e7: Deleting consensus metadata
I20260812 11:27:01.595371 2421 catalog_manager.cc:4841] TS b8df244fc61741e59f885648276269e7 (127.12.110.193:45097): tablet e4ac382643a24a5c93cf186fc50bd0db (table test-table [id=a31987a895744ec98aab4eb3a212caf9]) successfully deleted
I20260812 11:27:01.597254 3386 raft_consensus.cc:2229] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:01.597954 3386 raft_consensus.cc:2258] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:01.600101 3386 ts_tablet_manager.cc:1905] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:01.601438 3383 ts_tablet_manager.cc:1918] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:01.601682 3383 log.cc:1196] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/e4ac382643a24a5c93cf186fc50bd0db
I20260812 11:27:01.602385 3383 ts_tablet_manager.cc:1939] T e4ac382643a24a5c93cf186fc50bd0db P dd03439b6ded4fb99b5a0df47e19a0bc: Deleting consensus metadata
I20260812 11:27:01.604254 2417 catalog_manager.cc:4841] TS dd03439b6ded4fb99b5a0df47e19a0bc (127.12.110.194:43913): tablet e4ac382643a24a5c93cf186fc50bd0db (table test-table [id=a31987a895744ec98aab4eb3a212caf9]) successfully deleted
I20260812 11:27:01.606967 3386 ts_tablet_manager.cc:1918] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:01.607160 3386 log.cc:1196] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/e4ac382643a24a5c93cf186fc50bd0db
I20260812 11:27:01.607697 3386 ts_tablet_manager.cc:1939] T e4ac382643a24a5c93cf186fc50bd0db P d69b920832674b68beecdb748b7991ca: Deleting consensus metadata
I20260812 11:27:01.609468 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.583795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33512 (local address 127.12.110.193:45097)
0812 11:27:01.584042 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.584048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.591810 (+ 7762us) server_negotiation.cc:407] Connection header received
0812 11:27:01.591992 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.591998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.592086 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.592193 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:01.594119 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.595274 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.597224 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.597863 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.597960 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.601642 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:01.601655 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:01.601673 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:01.601713 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:01.605625 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:01.606513 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:01.606522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:01.606525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:01.606725 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:01.607239 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:01.607246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:01.607252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:01.607782 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:01.608185 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.608782 (+ 597us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.609054 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260812 11:27:01.609627 2420 catalog_manager.cc:4841] TS d69b920832674b68beecdb748b7991ca (127.12.110.195:43041): tablet e4ac382643a24a5c93cf186fc50bd0db (table test-table [id=a31987a895744ec98aab4eb3a212caf9]) successfully deleted
I20260812 11:27:01.610184 2786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:01.724078 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 2518
I20260812 11:27:01.756811 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 2807
I20260812 11:27:01.774092 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.757605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.12.110.194:43913)
0812 11:27:01.757803 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.757810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.757953 (+ 143us) server_negotiation.cc:407] Connection header received
0812 11:27:01.758125 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.758130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.758224 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.758329 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:01.760130 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.761272 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.763353 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.764044 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.764171 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.767604 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:01.767615 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:01.767630 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:01.767668 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:01.771252 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:01.772044 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:01.772050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:01.772053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:01.772201 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:01.772605 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:01.772609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:01.772611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:01.773020 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:01.773243 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.773635 (+ 392us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.773796 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:27:01.774868 3038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:01.888317 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 2807
I20260812 11:27:01.915580 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 3063
I20260812 11:27:01.929370 3315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:01.916405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.12.110.195:43041)
0812 11:27:01.916605 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:01.916611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:01.916631 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:27:01.916806 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:01.916811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:01.916899 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:01.917004 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:01.918343 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.919146 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.920461 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:01.920883 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:01.921017 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:01.923779 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:01.923794 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:01.923822 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:01.923853 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:01.926645 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:01.927326 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:01.927335 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:01.927338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:01.927471 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:01.927847 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:01.927852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:01.927855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:01.928225 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:01.928557 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:01.928839 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 11:27:01.929052 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:27:01.930061 3276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:02.020567 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 3063
I20260812 11:27:02.052925 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 2381
I20260812 11:27:02.054201 2507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:02.169395 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 2381
I20260812 11:27:02.203171 12731 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260812 11:27:02.548920 12731 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260812 11:27:02.549150 12731 mini_ranger.cc:66] Stopping Ranger...
I20260812 11:27:02.904654 12731 mini_ranger.cc:68] Stopped Ranger
2026-08-12 11:27:02.904 UTC [644] LOG: received smart shutdown request
2026-08-12 11:27:02.908 UTC [644] LOG: background worker "logical replication launcher" (PID 659) exited with exit code 1
2026-08-12 11:27:02.908 UTC [652] LOG: shutting down
2026-08-12 11:27:03.080 UTC [644] LOG: database system is shut down
2026-08-12T11:27:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57527 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:27:03 dist-test-slave-h6n0 krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:27:03 dist-test-slave-h6n0 krb5kdc[3410](info): set up 2 sockets
Aug 12 11:27:03 dist-test-slave-h6n0 krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260812 11:27:04.628785 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.366s user 0.004s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:27:04 dist-test-slave-h6n0 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:27:04.936771 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36737
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36737,127.12.110.253:39751,127.12.110.252:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:05.287722 3431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:05.288439 3431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:05.288702 3431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:05.297988 3431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:05.298180 3431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:05.298316 3431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:05.298442 3431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:05.298517 3431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:05.310416 3431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36737,127.12.110.253:39751,127.12.110.252:45019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:05.311964 3431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:05.313526 3431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:05.327499 3440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:05.327565 3438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:05.327468 3437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:05.329538 3431 server_base.cc:1034] running on GCE node
I20260812 11:27:05.330406 3431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:05.331961 3431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:05.333212 3431 hybrid_clock.cc:648] HybridClock initialized: now 1786534025333224 us; error 134 us; skew 500 ppm
I20260812 11:27:05.333606 3431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:05 dist-test-slave-h6n0 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:05.337148 3431 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:27:05.338847 3431 webserver.cc:460] Webserver started at http://127.12.110.254:43499/ using document root <none> and password file <none>
I20260812 11:27:05.339572 3431 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:05.339720 3431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:05.340035 3431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:05.343180 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "f5417c55b3a140eaaf0b52e9db68b7ee"
format_stamp: "Formatted at 2026-08-12 11:27:05 on dist-test-slave-h6n0"
server_key: "0c58b74b613652c1427162cd98ba9b22"
server_key_iv: "e7dbd8d0789e8f5b0ed1c5c8df6b28ab"
server_key_version: "encryptionkey@0"
I20260812 11:27:05.344000 3431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "f5417c55b3a140eaaf0b52e9db68b7ee"
format_stamp: "Formatted at 2026-08-12 11:27:05 on dist-test-slave-h6n0"
server_key: "0c58b74b613652c1427162cd98ba9b22"
server_key_iv: "e7dbd8d0789e8f5b0ed1c5c8df6b28ab"
server_key_version: "encryptionkey@0"
I20260812 11:27:05.349331 3431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 11:27:05.353258 3447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:05.354645 3431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:27:05.354835 3431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "f5417c55b3a140eaaf0b52e9db68b7ee"
format_stamp: "Formatted at 2026-08-12 11:27:05 on dist-test-slave-h6n0"
server_key: "0c58b74b613652c1427162cd98ba9b22"
server_key_iv: "e7dbd8d0789e8f5b0ed1c5c8df6b28ab"
server_key_version: "encryptionkey@0"
I20260812 11:27:05.355054 3431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:05.376838 3431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:05.381726 3431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:05.382032 3431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:05.403658 3431 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:36737
I20260812 11:27:05.403678 3528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:36737 every 8 connection(s)
I20260812 11:27:05.405164 3431 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:27:05.408898 3530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:05.414906 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 3431
I20260812 11:27:05.415246 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:27:05.416168 12731 external_mini_cluster.cc:1428] Setting key 26729d614b1c78eb685b48e7b290b108
I20260812 11:27:05.416299 3530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:27:05 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:05.445426 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.421233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36737 (local address 127.0.0.1:36560)
0812 11:27:05.422190 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.422239 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:05.424891 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:05.426261 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:05.426274 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:05.427119 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:05.427614 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:05.427662 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.429414 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.429420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:05.430493 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:05.430506 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.431492 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.431500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:05.431749 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:05.432442 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:05.432464 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:05.435356 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:05.440430 (+ 5074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:05.440442 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:05.440452 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:05.440878 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:05.441602 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:05.441611 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:05.441616 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:05.441950 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:05.443302 (+ 1352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:05.443309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:05.443528 (+ 219us) client_negotiation.cc:769] Sending connection context
0812 11:27:05.443767 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 11:27:05.444219 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":49,"thread_start_us":439,"threads_started":1}
I20260812 11:27:05.445472 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.421234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.12.110.254:36737)
0812 11:27:05.423045 (+ 1811us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:05.423054 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:05.423108 (+ 54us) server_negotiation.cc:407] Connection header received
0812 11:27:05.425500 (+ 2392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:05.425508 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:05.425877 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:05.426122 (+ 245us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:05.427891 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.429232 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.430671 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.431232 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.431430 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:05.435682 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:05.435698 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:05.435720 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:05.435769 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:05.440261 (+ 4492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:05.441051 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:05.441065 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:05.441071 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:05.441341 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:05.442160 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:05.442169 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:05.442174 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:05.443105 (+ 931us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:05.443359 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:05.443992 (+ 633us) server_negotiation.cc:299] Negotiation successful
0812 11:27:05.444423 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":1589,"spinlock_wait_cycles":17152,"thread_start_us":1359,"threads_started":1}
I20260812 11:27:05.449057 3530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.110.253@KRBTEST.COM; defaulting to no policy
I20260812 11:27:05.450203 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.449799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39364)
0812 11:27:05.449976 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.450096 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260812 11:27:05.450570 3451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
I20260812 11:27:05.452690 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.451923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39370)
0812 11:27:05.452422 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.452547 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.253@KRBTEST.COM" created.
W20260812 11:27:05.453073 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:39751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab.
Entry for principal kudu/127.12.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab.
I20260812 11:27:05.485929 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:39751
--webserver_interface=127.12.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab
--principal=kudu/127.12.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36737,127.12.110.253:39751,127.12.110.252:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:27:05.502656 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } attempt: 1
I20260812 11:27:05.503952 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.503494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39380)
0812 11:27:05.503694 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.503822 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 11:27:05.504930 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.504581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39384)
0812 11:27:05.504736 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.504802 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260812 11:27:05.505122 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:39751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
I20260812 11:27:05.573474 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } attempt: 2
I20260812 11:27:05.575210 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.574502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39400)
0812 11:27:05.574909 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.575019 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260812 11:27:05.576706 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.575995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39410)
0812 11:27:05.576333 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.576475 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:27:05.577126 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:39751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
I20260812 11:27:05.668471 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } attempt: 3
I20260812 11:27:05.669867 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.669362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39420)
0812 11:27:05.669580 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.669703 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 11:27:05.671504 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.671137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39422)
0812 11:27:05.671276 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.671352 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260812 11:27:05.671674 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:39751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
I20260812 11:27:05.814940 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } attempt: 4
I20260812 11:27:05.816145 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.815637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39424)
0812 11:27:05.815923 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.816014 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260812 11:27:05.817371 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.816736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39426)
0812 11:27:05.816883 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.817122 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 11:27:05.817790 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:39751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:39751: connect: Connection refused (error 111)
W20260812 11:27:05.837988 3538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:05.838265 3538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:05.838454 3538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:05.845691 3538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:05.845795 3538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:05.845898 3538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:05.845994 3538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:05.846078 3538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:05.857323 3538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36737,127.12.110.253:39751,127.12.110.252:45019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab
--principal=kudu/127.12.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:05.858844 3538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:05.860267 3538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:05.872890 3547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:05.872932 3551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:05.872993 3549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:05.873941 3538 server_base.cc:1034] running on GCE node
I20260812 11:27:05.874864 3538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:05.876236 3538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:05.877491 3538 hybrid_clock.cc:648] HybridClock initialized: now 1786534025877449 us; error 53 us; skew 500 ppm
I20260812 11:27:05.877846 3538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:05 dist-test-slave-h6n0 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:05.881677 3538 init.cc:377] Logged in from keytab as kudu/127.12.110.253@KRBTEST.COM (short username kudu)
I20260812 11:27:05.883631 3538 webserver.cc:460] Webserver started at http://127.12.110.253:45287/ using document root <none> and password file <none>
I20260812 11:27:05.884424 3538 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:05.884596 3538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:05.884944 3538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:05.888265 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data/instance:
uuid: "693859d162134c0890908753acfedfdb"
format_stamp: "Formatted at 2026-08-12 11:27:05 on dist-test-slave-h6n0"
server_key: "392201ee0ebc726234cbe956a051721e"
server_key_iv: "963347cbbc2d2ad053bd15917642bb94"
server_key_version: "encryptionkey@0"
I20260812 11:27:05.889168 3538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/wal/instance:
uuid: "693859d162134c0890908753acfedfdb"
format_stamp: "Formatted at 2026-08-12 11:27:05 on dist-test-slave-h6n0"
server_key: "392201ee0ebc726234cbe956a051721e"
server_key_iv: "963347cbbc2d2ad053bd15917642bb94"
server_key_version: "encryptionkey@0"
I20260812 11:27:05.895365 3538 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260812 11:27:05.899503 3559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:05.901548 3538 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:27:05.901780 3538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/wal
uuid: "693859d162134c0890908753acfedfdb"
format_stamp: "Formatted at 2026-08-12 11:27:05 on dist-test-slave-h6n0"
server_key: "392201ee0ebc726234cbe956a051721e"
server_key_iv: "963347cbbc2d2ad053bd15917642bb94"
server_key_version: "encryptionkey@0"
I20260812 11:27:05.901993 3538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:05.935943 3538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:05.942445 3538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:05.942834 3538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:05.965234 3538 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.253:39751
I20260812 11:27:05.965319 3658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.253:39751 every 8 connection(s)
I20260812 11:27:05.966713 3538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
I20260812 11:27:05.970420 3660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:05.976603 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 3538
I20260812 11:27:05.976827 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-1/wal/instance
I20260812 11:27:05.977275 12731 external_mini_cluster.cc:1428] Setting key 13082bc4249658481ee1c37c8a7b5834
I20260812 11:27:05.977162 3660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:27:05 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:06.001096 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.979850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36566 (local address 127.12.110.254:36737)
0812 11:27:05.980406 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:05.980416 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:05.981461 (+ 1045us) server_negotiation.cc:407] Connection header received
0812 11:27:05.983417 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:05.983426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:05.983521 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:05.983657 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:05.985102 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.986214 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.988419 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.989124 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.989259 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:05.993330 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:05.993340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:05.993348 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:05.993407 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:05.997554 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:05.998360 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:05.998369 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:05.998375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:05.998567 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:05.999036 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:05.999044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:05.999049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:05.999464 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:05.999622 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.000343 (+ 721us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.000603 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":271,"threads_started":1}
I20260812 11:27:06.001729 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:05.980443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36737 (local address 127.0.0.1:36566)
0812 11:27:05.981278 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:05.981325 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:05.983145 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:05.983842 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:05.983849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:05.984374 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:05.984915 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:05.984933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.986393 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.986401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:05.988248 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:05.988266 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:05.989364 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:05.989372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:05.989589 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:05.990313 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:05.990335 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:05.993079 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:05.997727 (+ 4648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:05.997734 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:05.997746 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:05.998197 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:05.998704 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:05.998708 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:05.998710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:05.998908 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:05.999598 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:05.999603 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:05.999921 (+ 318us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.000171 (+ 250us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.000574 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":370,"threads_started":1}
I20260812 11:27:06.003583 3660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.252@KRBTEST.COM" created.
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:27:06.021879 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.004643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39436)
0812 11:27:06.004809 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.004830 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:06.004946 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:06.006169 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:06.006178 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:06.006801 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:06.007021 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.007036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.008446 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.008451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.009369 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.009381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.011148 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.011161 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.011425 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.012121 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:06.012139 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:06.014182 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:06.019353 (+ 5171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.019357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.019359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.019740 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.020191 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.020195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.020197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.020341 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.021145 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:06.021149 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:06.021277 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.021399 (+ 122us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.021605 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:27:06.022357 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.004965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39436 (local address 127.12.110.253:39751)
0812 11:27:06.005729 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.005734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.005759 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:27:06.005801 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.005806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.005971 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.006087 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.007199 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.008309 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.009710 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.010957 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.011069 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.014424 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.014440 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.014461 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.014503 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.019213 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.019882 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.019893 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.019899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.020071 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.020449 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.020457 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.020463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.020996 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.021215 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.021567 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.021861 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":405,"threads_started":1}
I20260812 11:27:06.024797 3660 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:27:06.025964 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.025496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32858)
0812 11:27:06.025667 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.025808 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 11:27:06.026300 3564 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.027853 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.027534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32874)
0812 11:27:06.027693 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.027759 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:27:06.028174 3660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab.
Entry for principal kudu/127.12.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab.
I20260812 11:27:06.047974 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:45019
--webserver_interface=127.12.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab
--principal=kudu/127.12.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36737,127.12.110.253:39751,127.12.110.252:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:27:06.077565 3660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 1
I20260812 11:27:06.078951 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.078371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32890)
0812 11:27:06.078657 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.078777 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 11:27:06.080293 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.079616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32906)
0812 11:27:06.079789 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.079861 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 11:27:06.080727 3660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.117172 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } attempt: 5
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:27:06.133174 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.117905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39438)
0812 11:27:06.118154 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.118178 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:06.118313 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:06.118796 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:06.118802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:06.119327 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:06.119584 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.119597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.120640 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.120646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.121998 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.122015 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.123048 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.123055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.123283 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.123986 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:06.124009 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:06.126629 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:06.130642 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.130646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.130649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.131020 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.131539 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.131548 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.131553 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.131716 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.132376 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:06.132381 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:06.132521 (+ 140us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.132647 (+ 126us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.132884 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260812 11:27:06.133337 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.118235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.12.110.253:39751)
0812 11:27:06.118422 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.118427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.118445 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:27:06.118500 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.118504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.118638 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.118720 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.119759 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.120502 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.122195 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.122886 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.123020 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.126852 (+ 3832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.126869 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.126874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.126919 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.130451 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.131145 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.131151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.131154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.131315 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.131836 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.131841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.131844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.132226 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.132379 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.132787 (+ 408us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.133036 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:27:06.134461 3530 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:27:06.135565 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.135137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32914)
0812 11:27:06.135315 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.135411 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260812 11:27:06.136662 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.136243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32916)
0812 11:27:06.136412 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.136513 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:27:06.136930 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.148998 3660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 2
I20260812 11:27:06.149983 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.149587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32932)
0812 11:27:06.149761 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.149851 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260812 11:27:06.151023 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.150694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32946)
0812 11:27:06.150849 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.150914 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 11:27:06.151249 3660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.188208 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 1
I20260812 11:27:06.189654 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.189024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32952)
0812 11:27:06.189282 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.189425 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 11:27:06.191129 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.190699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32956)
0812 11:27:06.190869 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.190963 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:27:06.191423 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.242552 3660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 3
I20260812 11:27:06.243814 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.243361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32966)
0812 11:27:06.243558 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.243662 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260812 11:27:06.244892 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.244475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32976)
0812 11:27:06.244698 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.244773 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260812 11:27:06.245151 3660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.259706 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 2
I20260812 11:27:06.261140 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.260564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32978)
0812 11:27:06.260815 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.260909 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 11:27:06.262125 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.261805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32988)
0812 11:27:06.261952 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.262025 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260812 11:27:06.262327 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.368522 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 3
I20260812 11:27:06.369756 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.369180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:32990)
0812 11:27:06.369419 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.369535 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260812 11:27:06.370992 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.370493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:33002)
0812 11:27:06.370752 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.370850 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260812 11:27:06.371225 3530 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
I20260812 11:27:06.388389 3660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 4
I20260812 11:27:06.389519 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.389067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:33006)
0812 11:27:06.389291 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.389371 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 11:27:06.390434 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.390124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:33020)
0812 11:27:06.390268 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.390331 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260812 11:27:06.390669 3660 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:45019: Network error: Client connection negotiation failed: client connection to 127.12.110.252:45019: connect: Connection refused (error 111)
W20260812 11:27:06.413645 3676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:06.414014 3676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:06.414150 3676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:06.421932 3676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:06.422052 3676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:06.422101 3676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:06.422158 3676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:06.422204 3676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:06.432957 3676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36737,127.12.110.253:39751,127.12.110.252:45019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab
--principal=kudu/127.12.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:06.434198 3676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:06.435474 3676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:06.447739 3683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:06.447808 3687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:06.447768 3684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:06.448781 3676 server_base.cc:1034] running on GCE node
I20260812 11:27:06.449555 3676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:06.450947 3676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:06.452188 3676 hybrid_clock.cc:648] HybridClock initialized: now 1786534026452148 us; error 51 us; skew 500 ppm
I20260812 11:27:06.452550 3676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534026, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:06.455899 3676 init.cc:377] Logged in from keytab as kudu/127.12.110.252@KRBTEST.COM (short username kudu)
I20260812 11:27:06.457412 3676 webserver.cc:460] Webserver started at http://127.12.110.252:36525/ using document root <none> and password file <none>
I20260812 11:27:06.458088 3676 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:06.458230 3676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:06.458586 3676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:06.461553 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data/instance:
uuid: "d530598ae7d148c3898b11a1b3a0d612"
format_stamp: "Formatted at 2026-08-12 11:27:06 on dist-test-slave-h6n0"
server_key: "5e0dc0c3a9ffe871676440369d450bfd"
server_key_iv: "b23dcfa0921dfa44381f9c956a0b637e"
server_key_version: "encryptionkey@0"
I20260812 11:27:06.462340 3676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/wal/instance:
uuid: "d530598ae7d148c3898b11a1b3a0d612"
format_stamp: "Formatted at 2026-08-12 11:27:06 on dist-test-slave-h6n0"
server_key: "5e0dc0c3a9ffe871676440369d450bfd"
server_key_iv: "b23dcfa0921dfa44381f9c956a0b637e"
server_key_version: "encryptionkey@0"
I20260812 11:27:06.467344 3676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260812 11:27:06.471022 3699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:06.472349 3676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:27:06.472555 3676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/wal
uuid: "d530598ae7d148c3898b11a1b3a0d612"
format_stamp: "Formatted at 2026-08-12 11:27:06 on dist-test-slave-h6n0"
server_key: "5e0dc0c3a9ffe871676440369d450bfd"
server_key_iv: "b23dcfa0921dfa44381f9c956a0b637e"
server_key_version: "encryptionkey@0"
I20260812 11:27:06.472776 3676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:06.501091 3676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:06.507551 3676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:06.507900 3676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:06.530288 3676 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.252:45019
I20260812 11:27:06.530306 3789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.252:45019 every 8 connection(s)
I20260812 11:27:06.532420 3676 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
I20260812 11:27:06.537703 3793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:06.542337 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 3676
I20260812 11:27:06.542611 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786533802845873-12731-0/minicluster-data/master-2/wal/instance
I20260812 11:27:06.543104 12731 external_mini_cluster.cc:1428] Setting key 7427eae983d5c25b4d4e6a1cb76f21d7
I20260812 11:27:06.545699 3793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:27:06.548451 3530 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 4
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534026, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:06.570087 3533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.549067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:33030)
0812 11:27:06.549336 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.549370 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:06.549482 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:06.550573 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:06.550583 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:06.551094 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:06.551387 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.551404 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.554226 (+ 2822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.554234 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.555892 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.555909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.557174 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.557181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.557400 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.558123 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:06.558148 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:06.560895 (+ 2747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:06.565854 (+ 4959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.565867 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.565874 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.566792 (+ 918us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.567461 (+ 669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.567469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.567473 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.567680 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.568858 (+ 1178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:06.568864 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:06.569085 (+ 221us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.569339 (+ 254us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.569651 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260812 11:27:06.571870 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.549858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.12.110.254:36737)
0812 11:27:06.551799 (+ 1941us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.551807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.552899 (+ 1092us) server_negotiation.cc:407] Connection header received
0812 11:27:06.552979 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.552985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.553066 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.553175 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.554691 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.555431 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.557450 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.558123 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.558268 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.563191 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.563198 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.563202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.563249 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.567878 (+ 4629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.568736 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.568741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.568745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.568902 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.569388 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.569392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.569395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.569751 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.569904 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.570994 (+ 1090us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.571397 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1824,"thread_start_us":1764,"threads_started":1}
I20260812 11:27:06.572455 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.549878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36737 (local address 127.0.0.1:36572)
0812 11:27:06.552540 (+ 2662us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.552610 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:06.552778 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:06.553278 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:06.553286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:06.554135 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:06.554517 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.554569 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.555607 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.555612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.557269 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.557284 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.558372 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.558381 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.558620 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.559403 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:06.559440 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:06.562992 (+ 3552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:06.567998 (+ 5006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.568020 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.568028 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.568610 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.569057 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.569063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.569066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.569252 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.569978 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:06.569985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:06.570174 (+ 189us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.570390 (+ 216us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.571087 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2381,"mutex_wait_us":1,"thread_start_us":2129,"threads_started":1}
I20260812 11:27:06.572515 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.545024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.12.110.252:45019)
0812 11:27:06.545814 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.545827 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.545870 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:27:06.547856 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.547878 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.548305 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.548711 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.551984 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.554124 (+ 2140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.556204 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.556962 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.557189 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.562675 (+ 5486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.562686 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.562693 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.562741 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.566851 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.567824 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.567829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.567833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.567986 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.568447 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.568452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.568455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.568860 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.569168 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.570391 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.570641 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":224,"server-negotiator.queue_time_us":583,"thread_start_us":383,"threads_started":1}
I20260812 11:27:06.573097 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.549458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.12.110.252:45019)
0812 11:27:06.550163 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.550169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.550197 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:27:06.550247 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.550252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.550327 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.551575 (+ 1248us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.551907 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.554057 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.556067 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.556965 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.557717 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.561132 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.561141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.561154 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.561183 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.565603 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.567016 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.567026 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.567036 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.567282 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.567827 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.567832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.567835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.568659 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.568947 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.569653 (+ 706us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.570123 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":549,"spinlock_wait_cycles":25856,"thread_start_us":399,"threads_started":1}
I20260812 11:27:06.575953 3793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:27:06.581416 3530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee: Bootstrap starting.
I20260812 11:27:06.585902 3530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:06.587440 3530 log.cc:824] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:06.590883 3530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee: No bootstrap required, opened a new log
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534026, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:27:06.601676 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.577792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.12.110.254:36737)
0812 11:27:06.577967 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.577971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.577987 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:27:06.578030 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.578034 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.578127 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.578216 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.580124 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.581035 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.583083 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.583577 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.583679 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.589707 (+ 6028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.589714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.589719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.589759 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.596270 (+ 6511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.597700 (+ 1430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.597711 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.597715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.597900 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.598466 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.598473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.598478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.598931 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.599267 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.600863 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.601338 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:27:06.602226 3530 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.602700 3530 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:06.602828 3530 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5417c55b3a140eaaf0b52e9db68b7ee, State: Initialized, Role: FOLLOWER
I20260812 11:27:06.603134 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.578987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:39751 (local address 127.0.0.1:39452)
0812 11:27:06.579170 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.579189 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:06.579283 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:06.586806 (+ 7523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:06.586811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:06.587206 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:06.587360 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.587369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.588632 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.588639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.589951 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.589965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.590850 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.590857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.591068 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.591676 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:06.591703 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:06.594379 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:06.598676 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.598686 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.598690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.599487 (+ 797us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.600055 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.600063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.600068 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.600344 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.601399 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:06.601405 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:06.601602 (+ 197us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.601787 (+ 185us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.602754 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:27:06.603721 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.586249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39452 (local address 127.12.110.253:39751)
0812 11:27:06.586436 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.586440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.586458 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:27:06.586508 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.586511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.586625 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.586742 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.587517 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.588419 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.590109 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.590700 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.590854 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.594634 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.594641 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.594644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.594683 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.598515 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.599671 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.599677 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.599681 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.599868 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.600557 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.600565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.600569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.601119 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.601419 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.602796 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.603281 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:27:06.603634 3530 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.604740 3807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.605234 3793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:27:06.605376 3807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:27:06.606377 3530 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: configured and running, proceeding with master startup.
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534026, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:27:06.624528 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.605852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39456 (local address 127.12.110.253:39751)
0812 11:27:06.606055 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.606061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.606079 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:27:06.606130 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.606138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.606220 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.606386 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.608098 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.608829 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.610056 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.610507 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.610695 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.615967 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.615972 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.615975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.616008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.619651 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.620605 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.620614 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.620621 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.620859 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.621648 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.621656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.621662 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.622143 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.622605 (+ 462us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.623741 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.624078 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260812 11:27:06.625370 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.606271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33036 (local address 127.12.110.252:45019)
0812 11:27:06.606444 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.606450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.606522 (+ 72us) server_negotiation.cc:407] Connection header received
0812 11:27:06.606763 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.606767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.606843 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.606980 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.608020 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.609644 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.612217 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.612993 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.613133 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.617699 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.617709 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.617717 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.617766 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.621879 (+ 4113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.622864 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.622874 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.622880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.623092 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.623590 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.623595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.623598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.624095 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.624260 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.624878 (+ 618us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.625078 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:27:06.626443 3805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.605980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:33036)
0812 11:27:06.606150 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.606169 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:06.606629 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:06.607068 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:06.607073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:06.607560 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:06.607815 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.607832 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.610058 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.610068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.611938 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.611963 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.613176 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.613185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.613476 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.614211 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:06.614243 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:06.617442 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:06.622043 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.622051 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.622056 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.622655 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.623250 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.623256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.623260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.623455 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.624337 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:06.624342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:06.624530 (+ 188us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.625571 (+ 1041us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.626110 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 11:27:06.630895 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:27:06.632525 3793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: Bootstrap starting.
W20260812 11:27:06.635120 3454 tablet.cc:2369] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:27:06.635758 3822 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:27:06.636022 3822 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:27:06.638978 3793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:06.640723 3793 log.cc:824] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:06.644239 3793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: No bootstrap required, opened a new log
I20260812 11:27:06.656462 3793 raft_consensus.cc:348] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.656944 3793 raft_consensus.cc:374] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:06.657088 3793 raft_consensus.cc:729] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d530598ae7d148c3898b11a1b3a0d612, State: Initialized, Role: FOLLOWER
I20260812 11:27:06.642294 3431 master_runner.cc:186] Error getting master registration for 127.12.110.253:39751: OK, instance_id { permanent_uuid: "693859d162134c0890908753acfedfdb" instance_seqno: 1786534025959597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:27:06.657912 3793 consensus_queue.cc:260] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.658290 3431 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:27:06.659258 3833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.659780 3833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:27:06.660629 3793 sys_catalog.cc:564] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:27:06.689908 3660 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } attempt: 5
Aug 12 11:27:06 dist-test-slave-h6n0 krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534025, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:27:06.712039 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.690640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:45019 (local address 127.0.0.1:33060)
0812 11:27:06.690941 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:06.690963 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:06.691074 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:06.691633 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:06.691641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:06.692218 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:06.692447 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.692464 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.693999 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.694005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.695513 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:06.695534 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.696742 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.696749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:06.698622 (+ 1873us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.699337 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:06.699366 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:06.702638 (+ 3272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:06.708582 (+ 5944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.708590 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.708595 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.709201 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.709751 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:06.709757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:06.709761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:06.709984 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:06.710856 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:06.710862 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:06.711046 (+ 184us) client_negotiation.cc:769] Sending connection context
0812 11:27:06.711218 (+ 172us) client_negotiation.cc:240] Negotiation successful
0812 11:27:06.711535 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1559}
I20260812 11:27:06.713258 3797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.690834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33060 (local address 127.12.110.252:45019)
0812 11:27:06.691024 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.691030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.691051 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:27:06.691204 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.691210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.691295 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.691415 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.692653 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.693814 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.695708 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.696402 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.696529 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.702882 (+ 6353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.702890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.702895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.702939 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.708358 (+ 5419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.709404 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.709410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.709414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.709581 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.710144 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.710150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.710154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.710617 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.710789 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.712592 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.712860 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:27:06.717628 3670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.675382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39458 (local address 127.12.110.253:39751)
0812 11:27:06.675855 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.675867 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.679606 (+ 3739us) server_negotiation.cc:407] Connection header received
0812 11:27:06.679940 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.679949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.680049 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.680134 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.687407 (+ 7273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.689180 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.696215 (+ 7035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.696916 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.697017 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.706463 (+ 9446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.706474 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.706482 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.706547 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.710708 (+ 4161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.714461 (+ 3753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.714472 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.714475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.714865 (+ 390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.715472 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.715478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.715483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.715931 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.716210 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.716928 (+ 718us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.717209 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":79}
I20260812 11:27:06.721069 3660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: Bootstrap starting.
I20260812 11:27:06.723788 3804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.674144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36578 (local address 127.12.110.254:36737)
0812 11:27:06.674397 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.674407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.683185 (+ 8778us) server_negotiation.cc:407] Connection header received
0812 11:27:06.683516 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.683544 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.683666 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.683766 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.693052 (+ 9286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.694214 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.698426 (+ 4212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.699347 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.699467 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.711710 (+ 12243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.711719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.711725 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.711768 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.719798 (+ 8030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.720752 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.720760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.720763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.720943 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.721564 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.721571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.721574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.721977 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.722245 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.723008 (+ 763us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.723324 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 11:27:06.726148 3660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:06.727679 3660 log.cc:824] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:06.713583 3676 master_runner.cc:186] Error getting master registration for 127.12.110.253:39751: OK, instance_id { permanent_uuid: "693859d162134c0890908753acfedfdb" instance_seqno: 1786534025959597 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:27:06.742937 3676 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260812 11:27:06.740211 3861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:27:06.743148 3861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:27:06.746210 3660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: No bootstrap required, opened a new log
I20260812 11:27:06.752441 3803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:06.679730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.12.110.252:45019)
0812 11:27:06.679955 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:06.679960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:06.725373 (+ 45413us) server_negotiation.cc:407] Connection header received
0812 11:27:06.725655 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:06.725664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:06.725751 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:06.725849 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:06.735331 (+ 9482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.736939 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.739456 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:06.740301 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:06.740423 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:06.744226 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:06.744235 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:06.744246 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:06.744288 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:06.748246 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.749168 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.749174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.749178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.749354 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:06.749814 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:06.749819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:06.749822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:06.750190 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:06.750520 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:06.751761 (+ 1241us) server_negotiation.cc:299] Negotiation successful
0812 11:27:06.752008 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:27:06.758697 3660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.759166 3660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:06.759311 3660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 693859d162134c0890908753acfedfdb, State: Initialized, Role: FOLLOWER
I20260812 11:27:06.760031 3660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
W20260812 11:27:06.760545 3707 tablet.cc:2369] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:27:06.765494 3870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.767068 3660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:27:06.765889 3870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:27:06.780391 3807 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:27:06.780750 3807 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.782471 3807 leader_election.cc:288] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 693859d162134c0890908753acfedfdb (127.12.110.253:39751), d530598ae7d148c3898b11a1b3a0d612 (127.12.110.252:45019)
I20260812 11:27:06.783715 3743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d530598ae7d148c3898b11a1b3a0d612" is_pre_election: true
I20260812 11:27:06.784099 3610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693859d162134c0890908753acfedfdb" is_pre_election: true
I20260812 11:27:06.784417 3743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5417c55b3a140eaaf0b52e9db68b7ee in term 0.
I20260812 11:27:06.784657 3610 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5417c55b3a140eaaf0b52e9db68b7ee in term 0.
I20260812 11:27:06.785290 3452 leader_election.cc:302] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 693859d162134c0890908753acfedfdb, d530598ae7d148c3898b11a1b3a0d612, f5417c55b3a140eaaf0b52e9db68b7ee; no voters:
I20260812 11:27:06.785748 3807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:27:06.785916 3807 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:27:06.786037 3807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:06.786131 3538 master_runner.cc:263] No leader master found from master 693859d162134c0890908753acfedfdb
I20260812 11:27:06.786247 3538 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:27:06.790995 3807 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.791841 3807 leader_election.cc:288] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [CANDIDATE]: Term 1 election: Requested vote from peers 693859d162134c0890908753acfedfdb (127.12.110.253:39751), d530598ae7d148c3898b11a1b3a0d612 (127.12.110.252:45019)
I20260812 11:27:06.792208 3610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "693859d162134c0890908753acfedfdb"
I20260812 11:27:06.792378 3743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d530598ae7d148c3898b11a1b3a0d612"
I20260812 11:27:06.792436 3610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:06.792593 3743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:06.795878 3610 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5417c55b3a140eaaf0b52e9db68b7ee in term 1.
I20260812 11:27:06.796178 3743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5417c55b3a140eaaf0b52e9db68b7ee in term 1.
I20260812 11:27:06.796353 3451 leader_election.cc:302] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 693859d162134c0890908753acfedfdb, f5417c55b3a140eaaf0b52e9db68b7ee; no voters:
I20260812 11:27:06.797137 3807 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 1 FOLLOWER]: Leader election won for term 1
W20260812 11:27:06.800690 3889 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:27:06.800835 3889 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:27:06.800966 3807 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [term 1 LEADER]: Becoming Leader. State: Replica: f5417c55b3a140eaaf0b52e9db68b7ee, State: Running, Role: LEADER
I20260812 11:27:06.801581 3807 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } }
I20260812 11:27:06.820051 3891 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5417c55b3a140eaaf0b52e9db68b7ee. Latest consensus state: current_term: 1 leader_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.820591 3891 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:06.835201 3894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:27:06.838373 3894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:27:06.856841 3743 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [term 1 FOLLOWER]: Refusing update from remote peer f5417c55b3a140eaaf0b52e9db68b7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:06.859069 3891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:06.866000 3610 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [term 1 FOLLOWER]: Refusing update from remote peer f5417c55b3a140eaaf0b52e9db68b7ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:06.866958 3891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:06.905457 3833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5417c55b3a140eaaf0b52e9db68b7ee. Latest consensus state: current_term: 1 leader_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.905831 3833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:27:06.907181 3870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5417c55b3a140eaaf0b52e9db68b7ee. Latest consensus state: current_term: 1 leader_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.907536 3870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:27:06.909536 3895 mvcc.cc:205] Tried to move back new op lower bound from 7317643373987774464 to 7317643373784334336. Current Snapshot: MvccSnapshot[applied={T|T < 7317643373987774464}]
I20260812 11:27:06.920269 3896 mvcc.cc:205] Tried to move back new op lower bound from 7317643373987774464 to 7317643373784334336. Current Snapshot: MvccSnapshot[applied={T|T < 7317643373987774464}]
I20260812 11:27:06.936780 3897 mvcc.cc:205] Tried to move back new op lower bound from 7317643373987774464 to 7317643373784334336. Current Snapshot: MvccSnapshot[applied={T|T < 7317643373987774464}]
I20260812 11:27:06.949644 3833 sys_catalog.cc:455] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.949980 3833 sys_catalog.cc:458] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:27:06.954279 3807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.954876 3807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:06.955233 3807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.955776 3807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:06.957233 3870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5417c55b3a140eaaf0b52e9db68b7ee" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36737 } } peers { permanent_uuid: "693859d162134c0890908753acfedfdb" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 39751 } } peers { permanent_uuid: "d530598ae7d148c3898b11a1b3a0d612" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 45019 } } }
I20260812 11:27:06.957542 3870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:27:06.957885 3894 catalog_manager.cc:1347] Generated new cluster ID: 71abe83d264247238f8dc9ab31702297
I20260812 11:27:06.957998 3894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:27:06.983932 3894 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:27:06.986060 3894 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:27:07.010145 3894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5417c55b3a140eaaf0b52e9db68b7ee: Generated new TSK 0
I20260812 11:27:07.012403 3894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:27:07.745553 3861 catalog_manager.cc:1259] Loaded cluster ID: 71abe83d264247238f8dc9ab31702297
I20260812 11:27:07.745687 3861 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: loading cluster ID for follower catalog manager: success
I20260812 11:27:07.748493 3861 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: acquiring CA information for follower catalog manager: success
I20260812 11:27:07.751163 3861 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d530598ae7d148c3898b11a1b3a0d612: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:27:07.802875 3889 catalog_manager.cc:1259] Loaded cluster ID: 71abe83d264247238f8dc9ab31702297
I20260812 11:27:07.802981 3889 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: loading cluster ID for follower catalog manager: success
I20260812 11:27:07.805384 3889 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: acquiring CA information for follower catalog manager: success
I20260812 11:27:07.807339 3889 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 693859d162134c0890908753acfedfdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:27:08.167981 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 3431
I20260812 11:27:08.169173 3523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:08.286473 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 3431
I20260812 11:27:08.319936 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 3538
I20260812 11:27:08.321305 3648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:08.430454 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 3538
I20260812 11:27:08.458317 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 3676
I20260812 11:27:08.459874 3784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:08.570761 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 3676
2026-08-12T11:27:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5364 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:27:08 dist-test-slave-h6n0 krb5kdc[3922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:27:08 dist-test-slave-h6n0 krb5kdc[3922](info): set up 2 sockets
Aug 12 11:27:08 dist-test-slave-h6n0 krb5kdc[3922](info): commencing operation
krb5kdc: starting...
W20260812 11:27:09.975205 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.004s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:27:10 dist-test-slave-h6n0 krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:27:10.302714 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40901
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:38259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:40901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:10.667991 3944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:10.668313 3944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:10.668431 3944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:10.677147 3944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:10.677301 3944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:10.677354 3944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:10.677415 3944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:10.677465 3944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:10.688441 3944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:38259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:40901
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:40901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:10.689793 3944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:10.691840 3944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:10.706439 3950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:10.706393 3956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:10.706468 3952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:10.707949 3944 server_base.cc:1034] running on GCE node
I20260812 11:27:10.708997 3944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:10.710924 3944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:10.712194 3944 hybrid_clock.cc:648] HybridClock initialized: now 1786534030712152 us; error 53 us; skew 500 ppm
I20260812 11:27:10.712605 3944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:10 dist-test-slave-h6n0 krb5kdc[3922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534030, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:10.720621 3944 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:27:10.722223 3944 webserver.cc:460] Webserver started at http://127.12.110.254:37323/ using document root <none> and password file <none>
I20260812 11:27:10.723003 3944 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:10.723152 3944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:10.723481 3944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:10.726653 3944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "8cd1e8b7fe7a4389916c5897a476e422"
format_stamp: "Formatted at 2026-08-12 11:27:10 on dist-test-slave-h6n0"
server_key: "369a9633514a31b9b2e9f70692288142"
server_key_iv: "0487ff3616bcd99afc78dffbc842e60f"
server_key_version: "encryptionkey@0"
I20260812 11:27:10.727442 3944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "8cd1e8b7fe7a4389916c5897a476e422"
format_stamp: "Formatted at 2026-08-12 11:27:10 on dist-test-slave-h6n0"
server_key: "369a9633514a31b9b2e9f70692288142"
server_key_iv: "0487ff3616bcd99afc78dffbc842e60f"
server_key_version: "encryptionkey@0"
I20260812 11:27:10.732854 3944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:27:10.736500 3967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:10.737898 3944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:27:10.738109 3944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "8cd1e8b7fe7a4389916c5897a476e422"
format_stamp: "Formatted at 2026-08-12 11:27:10 on dist-test-slave-h6n0"
server_key: "369a9633514a31b9b2e9f70692288142"
server_key_iv: "0487ff3616bcd99afc78dffbc842e60f"
server_key_version: "encryptionkey@0"
I20260812 11:27:10.738330 3944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:10.753206 3944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:10.757959 3944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:10.758378 3944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:10.779266 3944 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:40901
I20260812 11:27:10.779299 4044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:40901 every 8 connection(s)
I20260812 11:27:10.780733 3944 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:27:10.784622 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 3944
I20260812 11:27:10.784672 4047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:10.784885 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:27:10.785624 12731 external_mini_cluster.cc:1428] Setting key 1cb0bc197b601b9398c3dd2cb802ab68
I20260812 11:27:10.799852 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422: Bootstrap starting.
Aug 12 11:27:10 dist-test-slave-h6n0 krb5kdc[3922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:10.804113 4047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:10.805220 4047 log.cc:824] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:10.808295 4047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422: No bootstrap required, opened a new log
I20260812 11:27:10.812063 4055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:10.787855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.12.110.254:40901)
0812 11:27:10.788806 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:10.788822 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:10.788874 (+ 52us) server_negotiation.cc:407] Connection header received
0812 11:27:10.791759 (+ 2885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:10.791792 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:10.792345 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:10.792850 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:10.794361 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:10.796466 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:10.798002 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:10.799078 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:10.799271 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:10.804495 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:10.804504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:10.804519 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:10.804561 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:10.807807 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:10.808653 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:10.808660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:10.808666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:10.808836 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:10.809254 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:10.809260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:10.809269 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:10.809801 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:10.810160 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:10.810651 (+ 491us) server_negotiation.cc:299] Negotiation successful
0812 11:27:10.811087 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":379,"threads_started":1}
I20260812 11:27:10.816179 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:27:10.817224 4047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd1e8b7fe7a4389916c5897a476e422" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40901 } }
I20260812 11:27:10.817617 4047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:10.817729 4047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd1e8b7fe7a4389916c5897a476e422, State: Initialized, Role: FOLLOWER
I20260812 11:27:10.818346 4047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd1e8b7fe7a4389916c5897a476e422" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40901 } }
I20260812 11:27:10.818635 4047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:27:10.818821 4047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:27:10.819041 4047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:10.821447 4047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd1e8b7fe7a4389916c5897a476e422" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40901 } }
I20260812 11:27:10.821944 4047 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd1e8b7fe7a4389916c5897a476e422; no voters:
I20260812 11:27:10.822577 4047 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:27:10.822736 4057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:27:10.823432 4057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [term 1 LEADER]: Becoming Leader. State: Replica: 8cd1e8b7fe7a4389916c5897a476e422, State: Running, Role: LEADER
I20260812 11:27:10.824030 4057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd1e8b7fe7a4389916c5897a476e422" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40901 } }
I20260812 11:27:10.824779 4047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:27:10.827183 4059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd1e8b7fe7a4389916c5897a476e422. Latest consensus state: current_term: 1 leader_uuid: "8cd1e8b7fe7a4389916c5897a476e422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd1e8b7fe7a4389916c5897a476e422" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40901 } } }
I20260812 11:27:10.827503 4058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd1e8b7fe7a4389916c5897a476e422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd1e8b7fe7a4389916c5897a476e422" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 40901 } } }
I20260812 11:27:10.827792 4059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:10.827824 4058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:10.831179 4068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:27:10.835959 4068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:27:10.844421 4068 catalog_manager.cc:1347] Generated new cluster ID: f54a39f0a7a7423e853d444a96a67dfb
I20260812 11:27:10.844549 4068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:27:10.868959 4068 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:27:10.870972 4068 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:27:10.886240 4068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd1e8b7fe7a4389916c5897a476e422: Generated new TSK 0
I20260812 11:27:10.887363 4068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:27:10.930145 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 3944
I20260812 11:27:10.931389 4040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:11.031936 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 3944
2026-08-12T11:27:11Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2459 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:27:11 dist-test-slave-h6n0 krb5kdc[4095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:27:11 dist-test-slave-h6n0 krb5kdc[4095](info): set up 2 sockets
Aug 12 11:27:11 dist-test-slave-h6n0 krb5kdc[4095](info): commencing operation
krb5kdc: starting...
W20260812 11:27:12.451208 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.365s user 0.007s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:27:12 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:27:12.764560 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37049
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:13.115672 4123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:13.115985 4123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:13.116191 4123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:13.123950 4123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:13.124065 4123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:13.124176 4123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:13.124279 4123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:13.124361 4123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:13.134922 4123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:13.137307 4123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:13.139256 4123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:13.152587 4131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:13.152732 4133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:13.152556 4135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:13.154022 4123 server_base.cc:1034] running on GCE node
I20260812 11:27:13.155465 4123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:13.156980 4123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:13.158223 4123 hybrid_clock.cc:648] HybridClock initialized: now 1786534033158166 us; error 30 us; skew 500 ppm
I20260812 11:27:13.158981 4123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:13 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:13.163103 4123 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:27:13.164808 4123 webserver.cc:460] Webserver started at http://127.12.110.254:43215/ using document root <none> and password file <none>
I20260812 11:27:13.165638 4123 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:13.165777 4123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:13.166041 4123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:13.169538 4123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "c8376d5b8434490e9e9715aef2d44306"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "f29e63b4bc95c586b8dd1484c3476ed9"
server_key_iv: "957b5dd3186bed7cb53b6763c4762adc"
server_key_version: "encryptionkey@0"
I20260812 11:27:13.170280 4123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "c8376d5b8434490e9e9715aef2d44306"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "f29e63b4bc95c586b8dd1484c3476ed9"
server_key_iv: "957b5dd3186bed7cb53b6763c4762adc"
server_key_version: "encryptionkey@0"
I20260812 11:27:13.175947 4123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260812 11:27:13.181267 4147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:13.182895 4123 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260812 11:27:13.183105 4123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "c8376d5b8434490e9e9715aef2d44306"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "f29e63b4bc95c586b8dd1484c3476ed9"
server_key_iv: "957b5dd3186bed7cb53b6763c4762adc"
server_key_version: "encryptionkey@0"
I20260812 11:27:13.183336 4123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:13.204361 4123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:13.208928 4123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:13.209229 4123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:13.231899 4123 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37049
I20260812 11:27:13.232264 4232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37049 every 8 connection(s)
I20260812 11:27:13.233402 4123 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:27:13.237061 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 4123
I20260812 11:27:13.237323 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:27:13.237425 4233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:13.238046 12731 external_mini_cluster.cc:1428] Setting key d8b4499e96bfefac92f73eaee96d44f3
I20260812 11:27:13.253489 4233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Bootstrap starting.
Aug 12 11:27:13 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:13.257747 4233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:13.258911 4233 log.cc:824] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:13.262123 4233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: No bootstrap required, opened a new log
I20260812 11:27:13.265859 4237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:13.240468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.12.110.254:37049)
0812 11:27:13.241748 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:13.241766 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:13.241831 (+ 65us) server_negotiation.cc:407] Connection header received
0812 11:27:13.244448 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:13.244476 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:13.244916 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:13.245255 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:13.247602 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:13.248906 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:13.250936 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:13.251620 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:13.251752 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:13.257490 (+ 5738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:13.257502 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:13.257525 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:13.257588 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:13.261571 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:13.262455 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:13.262473 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:13.262479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:13.262721 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:13.263136 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:13.263140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:13.263142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:13.263676 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:13.263915 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:13.264398 (+ 483us) server_negotiation.cc:299] Negotiation successful
0812 11:27:13.264774 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":960,"thread_start_us":603,"threads_started":1}
I20260812 11:27:13.270488 4233 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:13.270956 4233 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:13.271080 4233 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8376d5b8434490e9e9715aef2d44306, State: Initialized, Role: FOLLOWER
I20260812 11:27:13.271670 4233 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:13.271925 4233 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:27:13.272073 4233 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:27:13.272259 4233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:13.274382 4233 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:13.274964 4233 leader_election.cc:302] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8376d5b8434490e9e9715aef2d44306; no voters:
I20260812 11:27:13.275789 4233 leader_election.cc:288] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:27:13.276005 4240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:27:13.276794 4240 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 LEADER]: Becoming Leader. State: Replica: c8376d5b8434490e9e9715aef2d44306, State: Running, Role: LEADER
I20260812 11:27:13.277599 4240 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:13.278030 4233 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:27:13.281373 4243 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8376d5b8434490e9e9715aef2d44306. Latest consensus state: current_term: 1 leader_uuid: "c8376d5b8434490e9e9715aef2d44306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } } }
I20260812 11:27:13.281391 4242 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8376d5b8434490e9e9715aef2d44306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } } }
I20260812 11:27:13.281806 4243 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:13.281806 4242 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:13.291677 4254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:27:13.295785 4254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:27:13.303334 4254 catalog_manager.cc:1347] Generated new cluster ID: 6bba7cf2a4394ae89b13ddefed10374f
I20260812 11:27:13.303447 4254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:27:13.319466 4254 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:27:13.321256 4254 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:27:13.333997 4254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Generated new TSK 0
I20260812 11:27:13.335072 4254 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:27:13.406162 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37049
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:13.781415 4275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:13.781714 4275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:13.781831 4275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:13.789779 4275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:13.789892 4275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:13.790098 4275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:27:13.799928 4275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:13.801271 4275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:13.802682 4275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:13.817703 4289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:13.817883 4286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:13.818555 4291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:13.819209 4275 server_base.cc:1034] running on GCE node
I20260812 11:27:13.819828 4275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:13.820917 4275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:13.822194 4275 hybrid_clock.cc:648] HybridClock initialized: now 1786534033822082 us; error 101 us; skew 500 ppm
I20260812 11:27:13.822652 4275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:13 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:13.832743 4275 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:27:13.835968 4275 webserver.cc:460] Webserver started at http://127.12.110.193:45111/ using document root <none> and password file <none>
I20260812 11:27:13.837271 4275 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:13.837415 4275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:13.837736 4275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:13.841625 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "3134433d7489452ca30c4e706af4bc34"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "8b96abe3ad7c297bb0c5ad4cba636d46"
server_key_iv: "cce558edbe5bd0f1ebe3d12c642860ff"
server_key_version: "encryptionkey@0"
I20260812 11:27:13.842746 4275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "3134433d7489452ca30c4e706af4bc34"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "8b96abe3ad7c297bb0c5ad4cba636d46"
server_key_iv: "cce558edbe5bd0f1ebe3d12c642860ff"
server_key_version: "encryptionkey@0"
I20260812 11:27:13.854229 4275 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.008s
I20260812 11:27:13.859578 4300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:13.861310 4275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:27:13.861511 4275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "3134433d7489452ca30c4e706af4bc34"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "8b96abe3ad7c297bb0c5ad4cba636d46"
server_key_iv: "cce558edbe5bd0f1ebe3d12c642860ff"
server_key_version: "encryptionkey@0"
I20260812 11:27:13.861749 4275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:13.884029 4275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:13.889653 4275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:13.889966 4275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:13.891045 4275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:13.892547 4275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:13.892656 4275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:13.892818 4275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:13.892894 4275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:13.931599 4275 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:44989
I20260812 11:27:13.931660 4503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:44989 every 8 connection(s)
I20260812 11:27:13.932924 4275 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:27:13.939819 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 4275
I20260812 11:27:13.940068 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:27:13.941035 12731 external_mini_cluster.cc:1428] Setting key a1bc81c9875603519aef87669049476c
Aug 12 11:27:13 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534033, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:13.956413 4508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:13.935984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:57655 (local address 127.12.110.254:37049)
0812 11:27:13.936551 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:13.936556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:13.937298 (+ 742us) server_negotiation.cc:407] Connection header received
0812 11:27:13.939422 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:13.939427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:13.939495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:13.939593 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:13.941924 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:13.942811 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:13.944639 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:13.945376 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:13.945529 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:13.949350 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:13.949360 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:13.949368 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:13.949414 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:13.953029 (+ 3615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:13.953772 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:13.953781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:13.953787 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:13.953938 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:13.954407 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:13.954414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:13.954420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:13.954825 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:13.954981 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:13.955681 (+ 700us) server_negotiation.cc:299] Negotiation successful
0812 11:27:13.955930 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":322,"threads_started":1}
I20260812 11:27:13.957235 4510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:13.936424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:57655)
0812 11:27:13.937138 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:13.937183 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:13.939095 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:13.939903 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:13.939916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:13.940783 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:13.941699 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:13.941729 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:13.942999 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:13.943004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:13.944465 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:13.944479 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:13.945624 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:13.945629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:13.945843 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:13.946640 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:13.946671 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:13.949110 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:13.953177 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:13.953184 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:13.953196 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:13.953648 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:13.954077 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:13.954081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:13.954083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:13.954271 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:13.954980 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:13.954986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:13.955285 (+ 299us) client_negotiation.cc:769] Sending connection context
0812 11:27:13.955542 (+ 257us) client_negotiation.cc:240] Negotiation successful
0812 11:27:13.955938 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":280,"threads_started":1}
I20260812 11:27:13.958905 4506 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37049
I20260812 11:27:13.959228 4506 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:13.960152 4506 heartbeater.cc:511] Master 127.12.110.254:37049 requested a full tablet report, sending...
I20260812 11:27:13.962412 4174 ts_manager.cc:194] Registered new tserver with Master: 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989)
I20260812 11:27:13.963912 4174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:57655
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:27:14.016201 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37049
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:14.371341 4516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:14.371624 4516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:14.371783 4516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:14.379050 4516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:14.379155 4516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:14.379369 4516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:27:14.389901 4516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:14.391249 4516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:14.392552 4516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:14.405838 4529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:14.406052 4524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:14.406019 4525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:14.407184 4516 server_base.cc:1034] running on GCE node
I20260812 11:27:14.407712 4516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:14.408607 4516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:14.409891 4516 hybrid_clock.cc:648] HybridClock initialized: now 1786534034409823 us; error 80 us; skew 500 ppm
I20260812 11:27:14.410259 4516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:14 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:14.413692 4516 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:27:14.415298 4516 webserver.cc:460] Webserver started at http://127.12.110.194:44297/ using document root <none> and password file <none>
I20260812 11:27:14.415994 4516 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:14.416147 4516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:14.416461 4516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:14.419520 4516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "c740c3ffa1664c3d86e9d74840581d1a"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "d1cac2cdcd6bc6d817e6acb638dd3d8f"
server_key_iv: "4bac9a2177e40fe418ee424347690a00"
server_key_version: "encryptionkey@0"
I20260812 11:27:14.420799 4516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "c740c3ffa1664c3d86e9d74840581d1a"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "d1cac2cdcd6bc6d817e6acb638dd3d8f"
server_key_iv: "4bac9a2177e40fe418ee424347690a00"
server_key_version: "encryptionkey@0"
I20260812 11:27:14.427600 4516 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260812 11:27:14.431671 4541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:14.433211 4516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:27:14.433432 4516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "c740c3ffa1664c3d86e9d74840581d1a"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "d1cac2cdcd6bc6d817e6acb638dd3d8f"
server_key_iv: "4bac9a2177e40fe418ee424347690a00"
server_key_version: "encryptionkey@0"
I20260812 11:27:14.433671 4516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:14.448607 4516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:14.452773 4516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:14.453074 4516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:14.454039 4516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:14.455621 4516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:14.455729 4516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:14.455890 4516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:14.455991 4516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:14.494058 4516 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:33759
I20260812 11:27:14.494279 4743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:33759 every 8 connection(s)
I20260812 11:27:14.495488 4516 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:27:14.497138 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 4516
I20260812 11:27:14.497363 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:27:14.498016 12731 external_mini_cluster.cc:1428] Setting key fbe0e8e7e741ecf23dcc869c12f717a5
Aug 12 11:27:14 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:14.516999 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:14.498433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:60687 (local address 127.12.110.254:37049)
0812 11:27:14.499019 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:14.499027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:14.500028 (+ 1001us) server_negotiation.cc:407] Connection header received
0812 11:27:14.502104 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:14.502110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:14.502200 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:14.502325 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:14.504247 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:14.505034 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:14.506584 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:14.507019 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:14.507153 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:14.510680 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:14.510686 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:14.510689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:14.510720 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:14.513751 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:14.514462 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:14.514468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:14.514472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:14.514666 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:14.515233 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:14.515238 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:14.515242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:14.515534 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:14.515694 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:14.516361 (+ 667us) server_negotiation.cc:299] Negotiation successful
0812 11:27:14.516598 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":323,"threads_started":1}
I20260812 11:27:14.517903 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:14.498989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:60687)
0812 11:27:14.499843 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:14.499890 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:14.501830 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:14.502621 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:14.502629 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:14.503476 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:14.504061 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:14.504079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:14.505204 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:14.505208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:14.506418 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:14.506430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:14.507266 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:14.507273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:14.507502 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:14.508228 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:14.508249 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:14.510418 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:14.513896 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:14.513905 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:14.513920 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:14.514333 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:14.514778 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:14.514784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:14.514787 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:14.515087 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:14.515669 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:14.515677 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:14.515996 (+ 319us) client_negotiation.cc:769] Sending connection context
0812 11:27:14.516266 (+ 270us) client_negotiation.cc:240] Negotiation successful
0812 11:27:14.516790 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":385,"threads_started":1}
I20260812 11:27:14.519523 4744 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37049
I20260812 11:27:14.519824 4744 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:14.520550 4744 heartbeater.cc:511] Master 127.12.110.254:37049 requested a full tablet report, sending...
I20260812 11:27:14.522092 4174 ts_manager.cc:194] Registered new tserver with Master: c740c3ffa1664c3d86e9d74840581d1a (127.12.110.194:33759)
I20260812 11:27:14.523630 4174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:60687
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:27:14.569900 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37049
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:14.909106 4751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:14.909417 4751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:14.909523 4751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:14.916791 4751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:14.916908 4751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:14.917030 4751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:27:14.926898 4751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:14.928296 4751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:14.929646 4751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:14.943615 4759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:14.943720 4760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:14.943738 4765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:14.944648 4751 server_base.cc:1034] running on GCE node
I20260812 11:27:14.945216 4751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:14.946202 4751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:14.947412 4751 hybrid_clock.cc:648] HybridClock initialized: now 1786534034947407 us; error 116 us; skew 500 ppm
I20260812 11:27:14.947746 4751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:14 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:14.951193 4751 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:27:14.952662 4751 webserver.cc:460] Webserver started at http://127.12.110.195:42481/ using document root <none> and password file <none>
I20260812 11:27:14.953320 4751 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:14.953469 4751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:14.953768 4751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:14.956745 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "c03d318167eb45aaad2372df724a033f"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "e7c05aa963d71a9a686d3aaa59aa789d"
server_key_iv: "f5908f902a60c3983dd4fa6b86c6a018"
server_key_version: "encryptionkey@0"
I20260812 11:27:14.957525 4751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "c03d318167eb45aaad2372df724a033f"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "e7c05aa963d71a9a686d3aaa59aa789d"
server_key_iv: "f5908f902a60c3983dd4fa6b86c6a018"
server_key_version: "encryptionkey@0"
I20260812 11:27:14.962477 4751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260812 11:27:14.966162 4774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:14.968333 4751 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:27:14.968566 4751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "c03d318167eb45aaad2372df724a033f"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "e7c05aa963d71a9a686d3aaa59aa789d"
server_key_iv: "f5908f902a60c3983dd4fa6b86c6a018"
server_key_version: "encryptionkey@0"
I20260812 11:27:14.968865 4751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:14.969412 4506 heartbeater.cc:503] Master 127.12.110.254:37049 was elected leader, sending a full tablet report...
I20260812 11:27:14.986510 4751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:14.991171 4751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:14.991503 4751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:14.992560 4751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:14.993950 4751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:14.994028 4751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:14.994161 4751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:14.994217 4751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:15.030035 4751 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:38721
I20260812 11:27:15.030071 4990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:38721 every 8 connection(s)
I20260812 11:27:15.031431 4751 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:27:15.040453 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 4751
I20260812 11:27:15.040634 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:27:15.041085 12731 external_mini_cluster.cc:1428] Setting key cdea708349fd30b042471080738052b7
Aug 12 11:27:15 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534034, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:15.053270 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.034328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:59327 (local address 127.12.110.254:37049)
0812 11:27:15.035068 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.035074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.035904 (+ 830us) server_negotiation.cc:407] Connection header received
0812 11:27:15.037803 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.037809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.037894 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.038001 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:15.039498 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.040480 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.042296 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.042797 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.042939 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.047420 (+ 4481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:15.047426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:15.047430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:15.047462 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:15.050058 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.050903 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.050909 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.050913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.051050 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.051499 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.051504 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.051508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.051788 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:15.051962 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.052639 (+ 677us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.052871 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":396,"threads_started":1}
I20260812 11:27:15.054217 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.034906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:59327)
0812 11:27:15.035715 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:15.035775 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:15.037434 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:15.038193 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:15.038200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:15.038781 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:15.039336 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.039353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.040843 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.040848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.042124 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.042137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.043047 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.043057 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.043288 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.044019 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:15.044041 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:15.047186 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:15.050226 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:15.050235 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:15.050251 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:15.050769 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:15.051166 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:15.051171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:15.051175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:15.051371 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:15.051933 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:15.051941 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:15.052275 (+ 334us) client_negotiation.cc:769] Sending connection context
0812 11:27:15.052538 (+ 263us) client_negotiation.cc:240] Negotiation successful
0812 11:27:15.053076 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":353,"threads_started":1}
I20260812 11:27:15.055886 4991 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37049
I20260812 11:27:15.056290 4991 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:15.057221 4991 heartbeater.cc:511] Master 127.12.110.254:37049 requested a full tablet report, sending...
I20260812 11:27:15.058861 4172 ts_manager.cc:194] Registered new tserver with Master: c03d318167eb45aaad2372df724a033f (127.12.110.195:38721)
I20260812 11:27:15.059726 4172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:59327
I20260812 11:27:15.070362 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:27:15.090355 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.075740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.12.110.254:37049)
0812 11:27:15.075951 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.075956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.076430 (+ 474us) server_negotiation.cc:407] Connection header received
0812 11:27:15.076604 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.076609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.076689 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.076778 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:15.078309 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.079315 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.080723 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.081185 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.081299 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.084064 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:15.084070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:15.084074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:15.084107 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:15.087370 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.088141 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.088148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.088154 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.088302 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.088734 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.088740 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.088745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.089105 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:15.089387 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.089734 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.089971 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 11:27:15.095620 4172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:27:15.102491 4172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:27:15.138222 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.114880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:33759 (local address 127.0.0.1:50318)
0812 11:27:15.128494 (+ 13614us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:15.128519 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:15.128668 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:15.129833 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:15.129845 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:15.129872 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:15.130638 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.130651 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.132470 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.132475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.134962 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.134980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.137117 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.137123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.137316 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.137381 (+ 65us) client_negotiation.cc:769] Sending connection context
0812 11:27:15.137506 (+ 125us) client_negotiation.cc:240] Negotiation successful
0812 11:27:15.137727 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13451,"mutex_wait_us":69,"thread_start_us":14534,"threads_started":1}
I20260812 11:27:15.138355 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.114953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.12.110.193:44989)
0812 11:27:15.128167 (+ 13214us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.128178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.128226 (+ 48us) server_negotiation.cc:407] Connection header received
0812 11:27:15.128359 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.128370 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.128693 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.128922 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:15.130200 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.131747 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.135312 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.136249 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.136463 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.137655 (+ 1192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.137792 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.137980 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12953,"thread_start_us":12770,"threads_started":1}
I20260812 11:27:15.138717 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.116462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.12.110.195:38721)
0812 11:27:15.126607 (+ 10145us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.126620 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.129886 (+ 3266us) server_negotiation.cc:407] Connection header received
0812 11:27:15.130210 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.130217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.130428 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.130619 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:15.131434 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.133079 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.135372 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.137185 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.137367 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.137525 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.138138 (+ 613us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.138319 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9902,"thread_start_us":9682,"threads_started":1}
I20260812 11:27:15.138269 5012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.114661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44989 (local address 127.0.0.1:53692)
0812 11:27:15.125483 (+ 10822us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:15.125553 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:15.125777 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:15.129277 (+ 3500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:15.129284 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:15.129318 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:15.129941 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.129954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.131880 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.131886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.135167 (+ 3281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.135177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.136459 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.136465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.137518 (+ 1053us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.137557 (+ 39us) client_negotiation.cc:769] Sending connection context
0812 11:27:15.137648 (+ 91us) client_negotiation.cc:240] Negotiation successful
0812 11:27:15.137801 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10469,"mutex_wait_us":908,"thread_start_us":13490,"threads_started":1}
I20260812 11:27:15.139321 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.115397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.12.110.194:33759)
0812 11:27:15.129243 (+ 13846us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.129251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.129276 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:27:15.129375 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.129384 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.129607 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.130033 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:15.130822 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.132310 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.135150 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.136625 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.136786 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.136958 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.138759 (+ 1801us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.138954 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13695,"thread_start_us":13563,"threads_started":1}
I20260812 11:27:15.139509 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.115672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38721 (local address 127.0.0.1:51276)
0812 11:27:15.129758 (+ 14086us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:15.129775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:15.130006 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:15.130878 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:15.130883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:15.130907 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:15.131240 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.131250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.133229 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.133237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.135170 (+ 1933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.135185 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.137715 (+ 2530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.137720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.137908 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.137943 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:27:15.138975 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0812 11:27:15.139125 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13939,"mutex_wait_us":66,"thread_start_us":9294,"threads_started":1}
I20260812 11:27:15.142640 4380 tablet_service.cc:1467] Processing CreateTablet for tablet 8484cdad9e1840dd83a5e9ecab70f908 (DEFAULT_TABLE table=test-table [id=b61febb401aa4e1387984ebe945c0924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:15.142858 4870 tablet_service.cc:1467] Processing CreateTablet for tablet 8484cdad9e1840dd83a5e9ecab70f908 (DEFAULT_TABLE table=test-table [id=b61febb401aa4e1387984ebe945c0924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:15.142863 4621 tablet_service.cc:1467] Processing CreateTablet for tablet 8484cdad9e1840dd83a5e9ecab70f908 (DEFAULT_TABLE table=test-table [id=b61febb401aa4e1387984ebe945c0924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:15.144208 4380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8484cdad9e1840dd83a5e9ecab70f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:15.144232 4870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8484cdad9e1840dd83a5e9ecab70f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:15.144230 4621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8484cdad9e1840dd83a5e9ecab70f908. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:15.155742 5014 tablet_bootstrap.cc:492] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Bootstrap starting.
I20260812 11:27:15.158564 5015 tablet_bootstrap.cc:492] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Bootstrap starting.
I20260812 11:27:15.158907 5014 tablet_bootstrap.cc:654] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:15.160346 5014 log.cc:824] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:15.161214 5021 tablet_bootstrap.cc:492] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Bootstrap starting.
I20260812 11:27:15.162127 5015 tablet_bootstrap.cc:654] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:15.163544 5015 log.cc:824] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:15.164654 5021 tablet_bootstrap.cc:654] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:15.164939 5014 tablet_bootstrap.cc:492] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: No bootstrap required, opened a new log
I20260812 11:27:15.165238 5014 ts_tablet_manager.cc:1397] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260812 11:27:15.165711 5021 log.cc:824] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:15.166558 5015 tablet_bootstrap.cc:492] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: No bootstrap required, opened a new log
I20260812 11:27:15.166816 5015 ts_tablet_manager.cc:1397] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260812 11:27:15.168251 5021 tablet_bootstrap.cc:492] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: No bootstrap required, opened a new log
I20260812 11:27:15.168503 5021 ts_tablet_manager.cc:1397] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260812 11:27:15.176414 5021 raft_consensus.cc:348] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.176793 5021 raft_consensus.cc:374] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:15.176904 5021 raft_consensus.cc:729] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c740c3ffa1664c3d86e9d74840581d1a, State: Initialized, Role: FOLLOWER
I20260812 11:27:15.176934 5014 raft_consensus.cc:348] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.177420 5014 raft_consensus.cc:374] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:15.177537 5014 raft_consensus.cc:729] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3134433d7489452ca30c4e706af4bc34, State: Initialized, Role: FOLLOWER
I20260812 11:27:15.177567 5021 consensus_queue.cc:260] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.177873 5015 raft_consensus.cc:348] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.178359 5015 raft_consensus.cc:374] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:15.178184 5014 consensus_queue.cc:260] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.178560 5015 raft_consensus.cc:729] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03d318167eb45aaad2372df724a033f, State: Initialized, Role: FOLLOWER
I20260812 11:27:15.178849 4744 heartbeater.cc:503] Master 127.12.110.254:37049 was elected leader, sending a full tablet report...
I20260812 11:27:15.179471 5021 ts_tablet_manager.cc:1428] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260812 11:27:15.179453 5015 consensus_queue.cc:260] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.183720 4991 heartbeater.cc:503] Master 127.12.110.254:37049 was elected leader, sending a full tablet report...
I20260812 11:27:15.184863 5015 ts_tablet_manager.cc:1428] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260812 11:27:15.186731 5014 ts_tablet_manager.cc:1428] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Time spent starting tablet: real 0.021s user 0.018s sys 0.005s
I20260812 11:27:15.236961 5030 raft_consensus.cc:482] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:27:15.237286 5030 raft_consensus.cc:504] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.239377 5030 leader_election.cc:288] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989), c03d318167eb45aaad2372df724a033f (127.12.110.195:38721)
W20260812 11:27:15.247665 4745 tablet.cc:2369] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:27:15.248044 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.239859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:55533 (local address 127.12.110.193:44989)
0812 11:27:15.240096 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.240103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.241156 (+ 1053us) server_negotiation.cc:407] Connection header received
0812 11:27:15.241475 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.241484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.241569 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.241661 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:15.242712 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.243798 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.245768 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.247028 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.247176 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.247247 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.247626 (+ 379us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.247766 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:27:15.248929 5038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.239644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44989 (local address 127.12.110.194:55533)
0812 11:27:15.241050 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:15.241070 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:15.241277 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:15.241845 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:15.241852 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:15.241881 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:15.242463 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.242477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.243952 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.243956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.245607 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.245622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.247254 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.247260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.247421 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.247458 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:27:15.248308 (+ 850us) client_negotiation.cc:240] Negotiation successful
0812 11:27:15.248476 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1227,"mutex_wait_us":57,"thread_start_us":1425,"threads_started":1}
I20260812 11:27:15.249251 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.240328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:46505 (local address 127.12.110.195:38721)
0812 11:27:15.240502 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.240508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.241520 (+ 1012us) server_negotiation.cc:407] Connection header received
0812 11:27:15.241674 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.241680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.241772 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.241860 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:15.243965 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.245032 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.246862 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.248034 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.248216 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.248286 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.248848 (+ 562us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.248996 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:27:15.249327 5036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.239778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38721 (local address 127.12.110.194:46505)
0812 11:27:15.241407 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:15.241425 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:15.241548 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:15.241892 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:15.241895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:15.241917 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:15.242471 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.242479 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.245203 (+ 2724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.245208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.246708 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:15.246721 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.248459 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.248465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:15.248667 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.248699 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:27:15.248809 (+ 110us) client_negotiation.cc:240] Negotiation successful
0812 11:27:15.248961 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"mutex_wait_us":25,"thread_start_us":889,"threads_started":1}
I20260812 11:27:15.250031 4417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8484cdad9e1840dd83a5e9ecab70f908" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3134433d7489452ca30c4e706af4bc34" is_pre_election: true
I20260812 11:27:15.250319 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8484cdad9e1840dd83a5e9ecab70f908" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03d318167eb45aaad2372df724a033f" is_pre_election: true
I20260812 11:27:15.250602 4417 raft_consensus.cc:2454] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 0.
I20260812 11:27:15.250808 4903 raft_consensus.cc:2454] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 0.
I20260812 11:27:15.251353 4543 leader_election.cc:302] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3134433d7489452ca30c4e706af4bc34, c740c3ffa1664c3d86e9d74840581d1a; no voters:
I20260812 11:27:15.252208 5030 raft_consensus.cc:2781] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:27:15.252584 5030 raft_consensus.cc:482] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:27:15.252712 5030 raft_consensus.cc:3037] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:15.255448 5030 raft_consensus.cc:504] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.256050 5030 leader_election.cc:288] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 election: Requested vote from peers 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989), c03d318167eb45aaad2372df724a033f (127.12.110.195:38721)
I20260812 11:27:15.256613 4903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8484cdad9e1840dd83a5e9ecab70f908" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03d318167eb45aaad2372df724a033f"
I20260812 11:27:15.256613 4417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8484cdad9e1840dd83a5e9ecab70f908" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3134433d7489452ca30c4e706af4bc34"
I20260812 11:27:15.256836 4903 raft_consensus.cc:3037] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:15.256836 4417 raft_consensus.cc:3037] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:15.260159 4417 raft_consensus.cc:2454] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 1.
I20260812 11:27:15.260159 4903 raft_consensus.cc:2454] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 1.
I20260812 11:27:15.260597 4543 leader_election.cc:302] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3134433d7489452ca30c4e706af4bc34, c03d318167eb45aaad2372df724a033f, c740c3ffa1664c3d86e9d74840581d1a; no voters:
I20260812 11:27:15.260972 5030 raft_consensus.cc:2781] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:27:15.261467 5030 raft_consensus.cc:686] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 1 LEADER]: Becoming Leader. State: Replica: c740c3ffa1664c3d86e9d74840581d1a, State: Running, Role: LEADER
I20260812 11:27:15.261955 5030 consensus_queue.cc:237] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:15.266896 4172 catalog_manager.cc:5495] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to c740c3ffa1664c3d86e9d74840581d1a (127.12.110.194). New cstate: current_term: 1 leader_uuid: "c740c3ffa1664c3d86e9d74840581d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } health_report { overall_health: UNKNOWN } } }
W20260812 11:27:15.283243 4992 tablet.cc:2369] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:27:15.298365 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.292805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50334 (local address 127.12.110.194:33759)
0812 11:27:15.293024 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.293032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.293054 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:27:15.293341 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.293346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.293440 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.293597 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:27:15.293995 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.294815 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.296192 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.296603 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.296739 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.297230 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:27:15.297742 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:27:15.297911 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.297993 (+ 82us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.298075 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:27:15.307511 4417 raft_consensus.cc:1261] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Refusing update from remote peer c740c3ffa1664c3d86e9d74840581d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:15.307516 4903 raft_consensus.cc:1261] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Refusing update from remote peer c740c3ffa1664c3d86e9d74840581d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:15.308486 5030 consensus_queue.cc:1035] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:15.308842 5039 consensus_queue.cc:1035] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:15.321206 5044 mvcc.cc:205] Tried to move back new op lower bound from 7317643408608886784 to 7317643408436944896. Current Snapshot: MvccSnapshot[applied={T|T < 7317643408608886784}]
I20260812 11:27:15.335362 4174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:15.335680 4174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:15.339974 4174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Sending DeleteTablet for 3 replicas of tablet 8484cdad9e1840dd83a5e9ecab70f908
I20260812 11:27:15.341292 4621 tablet_service.cc:1514] Processing DeleteTablet for tablet 8484cdad9e1840dd83a5e9ecab70f908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:15 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:50318
I20260812 11:27:15.341388 4870 tablet_service.cc:1514] Processing DeleteTablet for tablet 8484cdad9e1840dd83a5e9ecab70f908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:15 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:51276
I20260812 11:27:15.341765 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 4275
I20260812 11:27:15.342409 4380 tablet_service.cc:1514] Processing DeleteTablet for tablet 8484cdad9e1840dd83a5e9ecab70f908 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:15 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:53692
I20260812 11:27:15.351186 5067 tablet_replica.cc:330] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: stopping tablet replica
I20260812 11:27:15.352639 5067 raft_consensus.cc:2229] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:15.354224 5067 raft_consensus.cc:2258] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:15.357744 5064 tablet_replica.cc:330] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: stopping tablet replica
I20260812 11:27:15.358917 5064 raft_consensus.cc:2229] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:15.359575 5064 raft_consensus.cc:2258] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:15.359725 5067 ts_tablet_manager.cc:1905] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:15.360340 5062 tablet_replica.cc:330] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: stopping tablet replica
I20260812 11:27:15.360740 5062 raft_consensus.cc:2229] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:27:15.361462 5062 raft_consensus.cc:2258] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:15.364805 5064 ts_tablet_manager.cc:1905] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:15.364996 5062 ts_tablet_manager.cc:1905] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:15.373247 5064 ts_tablet_manager.cc:1918] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:15.373533 5064 log.cc:1196] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/8484cdad9e1840dd83a5e9ecab70f908
I20260812 11:27:15.374199 5064 ts_tablet_manager.cc:1939] T 8484cdad9e1840dd83a5e9ecab70f908 P c03d318167eb45aaad2372df724a033f: Deleting consensus metadata
I20260812 11:27:15.374715 5062 ts_tablet_manager.cc:1918] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:15.375017 5062 log.cc:1196] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/8484cdad9e1840dd83a5e9ecab70f908
I20260812 11:27:15.375653 5062 ts_tablet_manager.cc:1939] T 8484cdad9e1840dd83a5e9ecab70f908 P c740c3ffa1664c3d86e9d74840581d1a: Deleting consensus metadata
I20260812 11:27:15.376421 4153 catalog_manager.cc:4841] TS c03d318167eb45aaad2372df724a033f (127.12.110.195:38721): tablet 8484cdad9e1840dd83a5e9ecab70f908 (table test-table [id=b61febb401aa4e1387984ebe945c0924]) successfully deleted
I20260812 11:27:15.377467 4148 catalog_manager.cc:4841] TS c740c3ffa1664c3d86e9d74840581d1a (127.12.110.194:33759): tablet 8484cdad9e1840dd83a5e9ecab70f908 (table test-table [id=b61febb401aa4e1387984ebe945c0924]) successfully deleted
I20260812 11:27:15.380056 5067 ts_tablet_manager.cc:1918] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:15.380265 5067 log.cc:1196] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/8484cdad9e1840dd83a5e9ecab70f908
I20260812 11:27:15.380872 5067 ts_tablet_manager.cc:1939] T 8484cdad9e1840dd83a5e9ecab70f908 P 3134433d7489452ca30c4e706af4bc34: Deleting consensus metadata
I20260812 11:27:15.382966 4150 catalog_manager.cc:4841] TS 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989): tablet 8484cdad9e1840dd83a5e9ecab70f908 (table test-table [id=b61febb401aa4e1387984ebe945c0924]) successfully deleted
Aug 12 11:27:15 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:27:15.394371 5009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.343275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.12.110.193:44989)
0812 11:27:15.343507 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.343516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.368816 (+ 25300us) server_negotiation.cc:407] Connection header received
0812 11:27:15.371883 (+ 3067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.371891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.372018 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.372106 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:15.374648 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.375905 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.377719 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.378416 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.378581 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.384838 (+ 6257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:15.384864 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:15.384884 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:15.384942 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:15.389957 (+ 5015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.391564 (+ 1607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.391574 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.391578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.391772 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.392334 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.392340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.392343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.392827 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:15.393160 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.393709 (+ 549us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.393960 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:27:15.395295 4496 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:15.491267 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 4275
I20260812 11:27:15.530052 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 4516
Aug 12 11:27:15 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:27:15.555503 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.530905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50350 (local address 127.12.110.194:33759)
0812 11:27:15.531098 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.531104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.531127 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:27:15.531293 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.531299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.531407 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.531532 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:15.532898 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.534119 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.535725 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.536496 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.536653 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.546169 (+ 9516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:15.546201 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:15.546230 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:15.546316 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:15.551464 (+ 5148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.552716 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.552726 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.552731 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.552956 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.553459 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.553465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.553468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.553989 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:15.554356 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.554837 (+ 481us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.555081 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:27:15.556705 4734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:15.677862 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 4516
I20260812 11:27:15.708120 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 4751
Aug 12 11:27:15 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:27:15.727670 5013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:15.709178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.12.110.195:38721)
0812 11:27:15.709357 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:15.709362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:15.709381 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:27:15.709484 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:15.709489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:15.709562 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:15.709645 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:15.710936 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.711715 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.713023 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:15.713411 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:15.713511 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:15.719351 (+ 5840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:15.719365 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:15.719383 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:15.719418 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:15.724746 (+ 5328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.725523 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.725530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.725536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.725703 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:15.726101 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:15.726106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:15.726108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:15.726464 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:15.726791 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:15.727135 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 11:27:15.727303 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:27:15.728331 4979 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:15.819255 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 4751
I20260812 11:27:15.843864 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 4123
I20260812 11:27:15.845199 4224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:15.956277 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 4123
I20260812 11:27:15.991006 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37049
--webserver_interface=127.12.110.254
--webserver_port=43215
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:16.358325 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:16.358717 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:16.358888 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:16.366230 5075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:16.366391 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:16.366497 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:16.366622 5075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:16.366693 5075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:16.377753 5075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=43215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:16.379122 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:16.380393 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:16.409312 5085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:16.409366 5088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:16.409348 5091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:16.409984 5075 server_base.cc:1034] running on GCE node
I20260812 11:27:16.411075 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:16.412665 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:16.413959 5075 hybrid_clock.cc:648] HybridClock initialized: now 1786534036413906 us; error 61 us; skew 500 ppm
I20260812 11:27:16.414435 5075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:16 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:16.418380 5075 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:27:16.420689 5075 webserver.cc:460] Webserver started at http://127.12.110.254:43215/ using document root <none> and password file <none>
I20260812 11:27:16.421880 5075 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:16.422106 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:16.428603 5075 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:27:16.431831 5103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:16.433163 5075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:27:16.433373 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "c8376d5b8434490e9e9715aef2d44306"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "f29e63b4bc95c586b8dd1484c3476ed9"
server_key_iv: "957b5dd3186bed7cb53b6763c4762adc"
server_key_version: "encryptionkey@0"
I20260812 11:27:16.434764 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:16.457183 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:16.463930 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:16.464197 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:16.486888 5075 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37049
I20260812 11:27:16.486930 5200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37049 every 8 connection(s)
I20260812 11:27:16.488497 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:27:16.493595 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 5075
I20260812 11:27:16.494669 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:44989
--local_ip_for_outbound_sockets=127.12.110.193
--tserver_master_addrs=127.12.110.254:37049
--webserver_port=45111
--webserver_interface=127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:27:16.497687 5204 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:27:16.503012 5204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Bootstrap starting.
I20260812 11:27:16.524438 5204 log.cc:824] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:16.537402 5204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:27:16.537977 5204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Bootstrap complete.
I20260812 11:27:16.548007 5204 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:16.548908 5204 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8376d5b8434490e9e9715aef2d44306, State: Initialized, Role: FOLLOWER
I20260812 11:27:16.549585 5204 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:16.549850 5204 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:27:16.549973 5204 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:27:16.550171 5204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:27:16.554178 5204 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:16.554792 5204 leader_election.cc:302] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8376d5b8434490e9e9715aef2d44306; no voters:
I20260812 11:27:16.555483 5204 leader_election.cc:288] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:27:16.555632 5209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:27:16.557474 5209 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 LEADER]: Becoming Leader. State: Replica: c8376d5b8434490e9e9715aef2d44306, State: Running, Role: LEADER
I20260812 11:27:16.558099 5209 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:16.558382 5204 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:27:16.561074 5210 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c8376d5b8434490e9e9715aef2d44306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } } }
I20260812 11:27:16.561585 5210 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:16.562935 5211 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8376d5b8434490e9e9715aef2d44306. Latest consensus state: current_term: 2 leader_uuid: "c8376d5b8434490e9e9715aef2d44306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } } }
I20260812 11:27:16.563547 5211 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:16.570281 5217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:27:16.579030 5217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:27:16.580679 5217 catalog_manager.cc:1259] Loaded cluster ID: 6bba7cf2a4394ae89b13ddefed10374f
I20260812 11:27:16.580897 5217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:27:16.585213 5217 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260812 11:27:16.604107 5217 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260812 11:27:16.626329 5217 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:27:16.628321 5217 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Loaded TSK: 0
I20260812 11:27:16.630033 5217 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:27:16.899019 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:16.899361 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:16.899546 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:16.907032 5206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:16.907202 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:16.907388 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:27:16.917284 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:44989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=45111
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:16.918598 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:16.919994 5206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:16.935256 5246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:16.935211 5250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:16.938858 5253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:16.940171 5206 server_base.cc:1034] running on GCE node
I20260812 11:27:16.940861 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:16.941839 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:16.943118 5206 hybrid_clock.cc:648] HybridClock initialized: now 1786534036943063 us; error 40 us; skew 500 ppm
I20260812 11:27:16.943480 5206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:16 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:16.947832 5206 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:27:16.949599 5206 webserver.cc:460] Webserver started at http://127.12.110.193:45111/ using document root <none> and password file <none>
I20260812 11:27:16.950387 5206 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:16.950590 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:16.957252 5206 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260812 11:27:16.960431 5268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:16.961968 5206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 11:27:16.962430 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "3134433d7489452ca30c4e706af4bc34"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "8b96abe3ad7c297bb0c5ad4cba636d46"
server_key_iv: "cce558edbe5bd0f1ebe3d12c642860ff"
server_key_version: "encryptionkey@0"
I20260812 11:27:16.964090 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:16.982641 5206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:16.986665 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:16.986938 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:16.987967 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:16.989487 5206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:16.989588 5206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:16.989665 5206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:16.989789 5206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:17.029242 5206 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:44989
I20260812 11:27:17.029486 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:44989 every 8 connection(s)
I20260812 11:27:17.030936 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:27:17.039180 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 5206
I20260812 11:27:17.040155 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:33759
--local_ip_for_outbound_sockets=127.12.110.194
--tserver_master_addrs=127.12.110.254:37049
--webserver_port=44297
--webserver_interface=127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:27:17 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534036, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:17.057478 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:17.034850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:55753 (local address 127.12.110.254:37049)
0812 11:27:17.035539 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:17.035559 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:17.036121 (+ 562us) server_negotiation.cc:407] Connection header received
0812 11:27:17.038634 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:17.038648 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:17.039061 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:17.039430 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:17.041096 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.042334 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.044476 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.045108 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.045317 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:17.049012 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:17.049020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:17.049032 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:17.049068 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:17.052643 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:17.053455 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:17.053462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:17.053467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:17.053644 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:17.054103 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:17.054107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:17.054109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:17.054629 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:17.054825 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:17.055812 (+ 987us) server_negotiation.cc:299] Negotiation successful
0812 11:27:17.056128 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":327,"threads_started":1}
I20260812 11:27:17.057525 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:17.035164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:55753)
0812 11:27:17.035935 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:17.035980 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:17.037863 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:17.039582 (+ 1719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:17.039596 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:17.040251 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:17.040858 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:17.040875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.042496 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.042504 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:17.044222 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:17.044235 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.045364 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.045372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:17.045638 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:17.046357 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:17.046380 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:17.048734 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:17.052823 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:17.052832 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:17.052849 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:17.053282 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:17.053767 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:17.053772 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:17.053776 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:17.053968 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:17.054775 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:17.054784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:17.055216 (+ 432us) client_negotiation.cc:769] Sending connection context
0812 11:27:17.055574 (+ 358us) client_negotiation.cc:240] Negotiation successful
0812 11:27:17.056273 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":351,"threads_started":1}
I20260812 11:27:17.060523 5459 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37049
I20260812 11:27:17.060850 5459 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:17.061582 5459 heartbeater.cc:511] Master 127.12.110.254:37049 requested a full tablet report, sending...
I20260812 11:27:17.064199 5131 ts_manager.cc:194] Registered new tserver with Master: 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989)
I20260812 11:27:17.065793 5131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:55753
W20260812 11:27:17.394893 5466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:17.395241 5466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:17.395445 5466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:17.403620 5466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:17.403789 5466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:17.403967 5466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:27:17.415570 5466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:33759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=44297
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:17.417008 5466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:17.418401 5466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:17.432261 5479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:17.432329 5474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:17.432305 5472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:17.433221 5466 server_base.cc:1034] running on GCE node
I20260812 11:27:17.433876 5466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:17.434973 5466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:17.436285 5466 hybrid_clock.cc:648] HybridClock initialized: now 1786534037436173 us; error 91 us; skew 500 ppm
I20260812 11:27:17.436883 5466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:17 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:17.441385 5466 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:27:17.443336 5466 webserver.cc:460] Webserver started at http://127.12.110.194:44297/ using document root <none> and password file <none>
I20260812 11:27:17.444003 5466 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:17.444137 5466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:17.450263 5466 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:27:17.453199 5493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:17.455176 5466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 11:27:17.455322 5466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "c740c3ffa1664c3d86e9d74840581d1a"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "d1cac2cdcd6bc6d817e6acb638dd3d8f"
server_key_iv: "4bac9a2177e40fe418ee424347690a00"
server_key_version: "encryptionkey@0"
I20260812 11:27:17.456324 5466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:17.475245 5466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:17.480914 5466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:17.481225 5466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:17.482288 5466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:17.483692 5466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:17.483784 5466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:17.483935 5466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:17.484015 5466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:17.519734 5466 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:33759
I20260812 11:27:17.519827 5701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:33759 every 8 connection(s)
I20260812 11:27:17.521198 5466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:27:17.528245 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 5466
I20260812 11:27:17.529034 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:38721
--local_ip_for_outbound_sockets=127.12.110.195
--tserver_master_addrs=127.12.110.254:37049
--webserver_port=42481
--webserver_interface=127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:27:17 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:17.543529 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:17.523782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:35129 (local address 127.12.110.254:37049)
0812 11:27:17.524029 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:17.524036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:17.525136 (+ 1100us) server_negotiation.cc:407] Connection header received
0812 11:27:17.526982 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:17.526987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:17.527067 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:17.527157 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:17.528721 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.529763 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.531719 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.532438 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.532583 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:17.536526 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:17.536535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:17.536543 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:17.536592 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:17.540258 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:17.540958 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:17.540966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:17.540973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:17.541135 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:17.541547 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:17.541554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:17.541560 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:17.541954 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:17.542112 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:17.543077 (+ 965us) server_negotiation.cc:299] Negotiation successful
0812 11:27:17.543263 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:27:17.545010 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:17.524243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:35129)
0812 11:27:17.524985 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:17.525022 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:17.526722 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:17.527369 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:17.527376 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:17.527881 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:17.528512 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:17.528530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.530269 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.530273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:17.531556 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:17.531570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:17.532671 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:17.532676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:17.532917 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:17.533748 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:17.533771 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:17.536296 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:17.540401 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:17.540408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:17.540419 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:17.540827 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:17.541253 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:17.541256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:17.541259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:17.541432 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:17.542107 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:17.542114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:17.542628 (+ 514us) client_negotiation.cc:769] Sending connection context
0812 11:27:17.542965 (+ 337us) client_negotiation.cc:240] Negotiation successful
0812 11:27:17.543550 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":314,"threads_started":1}
I20260812 11:27:17.546588 5702 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37049
I20260812 11:27:17.546929 5702 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:17.547818 5702 heartbeater.cc:511] Master 127.12.110.254:37049 requested a full tablet report, sending...
I20260812 11:27:17.549432 5131 ts_manager.cc:194] Registered new tserver with Master: c740c3ffa1664c3d86e9d74840581d1a (127.12.110.194:33759)
I20260812 11:27:17.550258 5131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:35129
W20260812 11:27:17.885288 5708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:17.885685 5708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:17.885869 5708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:17.893123 5708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:17.893273 5708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:17.893463 5708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:27:17.903091 5708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:38721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=42481
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:17.904392 5708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:17.905699 5708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:17.919867 5714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:17.920303 5716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:17.920303 5719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:17.921058 5708 server_base.cc:1034] running on GCE node
I20260812 11:27:17.921679 5708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:17.922752 5708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:17.924037 5708 hybrid_clock.cc:648] HybridClock initialized: now 1786534037923957 us; error 60 us; skew 500 ppm
I20260812 11:27:17.924398 5708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:17 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:17.927776 5708 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:27:17.929225 5708 webserver.cc:460] Webserver started at http://127.12.110.195:42481/ using document root <none> and password file <none>
I20260812 11:27:17.929872 5708 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:17.929986 5708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:17.936144 5708 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:27:17.939172 5738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:17.940568 5708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:27:17.940774 5708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "c03d318167eb45aaad2372df724a033f"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "e7c05aa963d71a9a686d3aaa59aa789d"
server_key_iv: "f5908f902a60c3983dd4fa6b86c6a018"
server_key_version: "encryptionkey@0"
I20260812 11:27:17.941780 5708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:17.962239 5708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:17.966378 5708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:17.966679 5708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:17.967701 5708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:17.969169 5708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:17.969305 5708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:17.969455 5708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:17.969555 5708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:18.005239 5708 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:38721
I20260812 11:27:18.005332 5954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:38721 every 8 connection(s)
I20260812 11:27:18.006809 5708 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:27:18.016222 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 5708
Aug 12 11:27:18 dist-test-slave-h6n0 krb5kdc[4095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534037, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:18.028151 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.009460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:34065 (local address 127.12.110.254:37049)
0812 11:27:18.009652 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.009657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.010878 (+ 1221us) server_negotiation.cc:407] Connection header received
0812 11:27:18.012727 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.012733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.012821 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.012935 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:18.014387 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.015199 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.017303 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.017980 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.018111 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.021779 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:18.021785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:18.021790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:18.021829 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:18.024835 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.025530 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.025538 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.025543 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.025671 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.026091 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.026096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.026100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.026497 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:18.026676 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.027484 (+ 808us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.027727 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:27:18.029063 5959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.009955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:34065)
0812 11:27:18.010717 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:18.010766 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:18.012454 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:18.013183 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:18.013191 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:18.013693 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:18.014220 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.014236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.015374 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.015378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.017143 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.017160 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.018227 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.018236 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.018481 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.019230 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:18.019257 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:18.021569 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:18.024969 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:18.024977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:18.024992 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:18.025400 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:18.025783 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:18.025788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:18.025791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:18.025978 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:18.026669 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:18.026675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:18.027048 (+ 373us) client_negotiation.cc:769] Sending connection context
0812 11:27:18.027357 (+ 309us) client_negotiation.cc:240] Negotiation successful
0812 11:27:18.027717 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":286,"threads_started":1}
I20260812 11:27:18.030494 5957 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37049
I20260812 11:27:18.030812 5957 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:18.031678 5957 heartbeater.cc:511] Master 127.12.110.254:37049 requested a full tablet report, sending...
I20260812 11:27:18.032874 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.017278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52794 (local address 127.12.110.254:37049)
0812 11:27:18.017872 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.017879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.018204 (+ 325us) server_negotiation.cc:407] Connection header received
0812 11:27:18.018409 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.018413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.018469 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.018613 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:18.020118 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.021226 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.022699 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.023430 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.023594 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.026273 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:18.026279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:18.026283 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:18.026313 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:18.029732 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.030673 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.030679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.030682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.030822 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.031293 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.031299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.031302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.031651 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:18.031927 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.032275 (+ 348us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.032496 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":302,"threads_started":1}
I20260812 11:27:18.033389 5131 ts_manager.cc:194] Registered new tserver with Master: c03d318167eb45aaad2372df724a033f (127.12.110.195:38721)
I20260812 11:27:18.034487 5131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:34065
I20260812 11:27:18.035621 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:27:18.042287 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.036635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.12.110.254:37049)
0812 11:27:18.036807 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.036813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.037336 (+ 523us) server_negotiation.cc:407] Connection header received
0812 11:27:18.037469 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.037474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.037591 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.037701 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:27:18.038096 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.039064 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.040524 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.040941 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.041036 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.041452 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:27:18.041844 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:27:18.041942 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.042065 (+ 123us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.042122 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:27:18.043344 5129 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:27:18.050735 5129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:27:18.069914 5459 heartbeater.cc:503] Master 127.12.110.254:37049 was elected leader, sending a full tablet report...
I20260812 11:27:18.077561 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.064405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:33759 (local address 127.0.0.1:50360)
0812 11:27:18.065783 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:18.065837 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:18.066039 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:18.067193 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:18.067200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:18.067242 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:18.067745 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.067760 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.070147 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.070153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.072926 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.072947 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.076286 (+ 3339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.076296 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.076565 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.076642 (+ 77us) client_negotiation.cc:769] Sending connection context
0812 11:27:18.076803 (+ 161us) client_negotiation.cc:240] Negotiation successful
0812 11:27:18.077054 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"threads_started":1}
I20260812 11:27:18.078576 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.064614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.12.110.194:33759)
0812 11:27:18.065253 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.065265 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.066282 (+ 1017us) server_negotiation.cc:407] Connection header received
0812 11:27:18.066420 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.066431 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.066718 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.066908 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:18.067965 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.069959 (+ 1994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.073142 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.075728 (+ 2586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.075881 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.076089 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.077865 (+ 1776us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.078112 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":345,"threads_started":1}
I20260812 11:27:18.085142 5583 tablet_service.cc:1467] Processing CreateTablet for tablet 136f3dc84c7b49eb9d7889405d1f04ac (DEFAULT_TABLE table=test-table [id=ab195f46bdaf442f8be7e76d13637482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:18.086772 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136f3dc84c7b49eb9d7889405d1f04ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:18.086810 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.066087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38721 (local address 127.0.0.1:51290)
0812 11:27:18.078942 (+ 12855us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:18.078966 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:18.079095 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:18.081214 (+ 2119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:18.081301 (+ 87us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:18.081332 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:18.081521 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.081533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.083446 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.083452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.084618 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.084630 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.085895 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.085901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.086113 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.086146 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:27:18.086277 (+ 131us) client_negotiation.cc:240] Negotiation successful
0812 11:27:18.086415 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12714,"mutex_wait_us":3,"thread_start_us":8614,"threads_started":1}
I20260812 11:27:18.086980 5981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.064918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44989 (local address 127.0.0.1:53712)
0812 11:27:18.074978 (+ 10060us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:18.074998 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:18.075116 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:18.079668 (+ 4552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:18.079676 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:18.079709 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:18.080036 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.080049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.082060 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.082065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.084149 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.084165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.086070 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.086077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.086297 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.086339 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:27:18.086422 (+ 83us) client_negotiation.cc:240] Negotiation successful
0812 11:27:18.086572 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9912,"mutex_wait_us":1,"thread_start_us":927,"threads_started":1}
I20260812 11:27:18.086838 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.066762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.12.110.195:38721)
0812 11:27:18.080062 (+ 13300us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.080070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.080102 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:27:18.080199 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.080206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.080832 (+ 626us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.081736 (+ 904us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:18.082072 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.083280 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.084784 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.085730 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.085996 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.086183 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.086316 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.086466 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13109,"thread_start_us":12880,"threads_started":1}
I20260812 11:27:18.087968 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.065668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53712 (local address 127.12.110.193:44989)
0812 11:27:18.078874 (+ 13206us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.078885 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.078916 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:27:18.079042 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.079056 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.079401 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.079661 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:18.080258 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.081887 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.084379 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.085798 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.087255 (+ 1457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.087426 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.087496 (+ 70us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.087647 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13017,"thread_start_us":14188,"threads_started":1}
I20260812 11:27:18.091787 5343 tablet_service.cc:1467] Processing CreateTablet for tablet 136f3dc84c7b49eb9d7889405d1f04ac (DEFAULT_TABLE table=test-table [id=ab195f46bdaf442f8be7e76d13637482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:18.091996 5838 tablet_service.cc:1467] Processing CreateTablet for tablet 136f3dc84c7b49eb9d7889405d1f04ac (DEFAULT_TABLE table=test-table [id=ab195f46bdaf442f8be7e76d13637482]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:18.093021 5343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136f3dc84c7b49eb9d7889405d1f04ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:18.093290 5838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 136f3dc84c7b49eb9d7889405d1f04ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:18.102730 5984 tablet_bootstrap.cc:492] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Bootstrap starting.
I20260812 11:27:18.105026 5989 tablet_bootstrap.cc:492] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Bootstrap starting.
I20260812 11:27:18.106328 5984 tablet_bootstrap.cc:654] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:18.108314 5991 tablet_bootstrap.cc:492] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Bootstrap starting.
I20260812 11:27:18.108650 5989 tablet_bootstrap.cc:654] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:18.109532 5984 log.cc:824] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:18.109721 5989 log.cc:824] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:18.111886 5991 tablet_bootstrap.cc:654] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:18.112766 5984 tablet_bootstrap.cc:492] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: No bootstrap required, opened a new log
I20260812 11:27:18.112857 5991 log.cc:824] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:18.112975 5989 tablet_bootstrap.cc:492] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: No bootstrap required, opened a new log
I20260812 11:27:18.113082 5984 ts_tablet_manager.cc:1397] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260812 11:27:18.113265 5989 ts_tablet_manager.cc:1397] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260812 11:27:18.116012 5991 tablet_bootstrap.cc:492] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: No bootstrap required, opened a new log
I20260812 11:27:18.116273 5991 ts_tablet_manager.cc:1397] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260812 11:27:18.123503 5991 raft_consensus.cc:348] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.123862 5991 raft_consensus.cc:374] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:18.123997 5991 raft_consensus.cc:729] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03d318167eb45aaad2372df724a033f, State: Initialized, Role: FOLLOWER
I20260812 11:27:18.123936 5984 raft_consensus.cc:348] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.123936 5989 raft_consensus.cc:348] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.124372 5984 raft_consensus.cc:374] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:18.124344 5989 raft_consensus.cc:374] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:18.124473 5989 raft_consensus.cc:729] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3134433d7489452ca30c4e706af4bc34, State: Initialized, Role: FOLLOWER
I20260812 11:27:18.124491 5984 raft_consensus.cc:729] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c740c3ffa1664c3d86e9d74840581d1a, State: Initialized, Role: FOLLOWER
I20260812 11:27:18.124661 5991 consensus_queue.cc:260] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.125047 5984 consensus_queue.cc:260] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.125047 5989 consensus_queue.cc:260] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.125896 5957 heartbeater.cc:503] Master 127.12.110.254:37049 was elected leader, sending a full tablet report...
I20260812 11:27:18.126571 5991 ts_tablet_manager.cc:1428] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Time spent starting tablet: real 0.010s user 0.005s sys 0.005s
I20260812 11:27:18.128633 5989 ts_tablet_manager.cc:1428] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260812 11:27:18.128888 5984 ts_tablet_manager.cc:1428] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Time spent starting tablet: real 0.016s user 0.004s sys 0.011s
I20260812 11:27:18.129415 5702 heartbeater.cc:503] Master 127.12.110.254:37049 was elected leader, sending a full tablet report...
I20260812 11:27:18.229962 6004 raft_consensus.cc:482] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:27:18.230266 6004 raft_consensus.cc:504] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.232172 6004 leader_election.cc:288] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989), c03d318167eb45aaad2372df724a033f (127.12.110.195:38721)
I20260812 11:27:18.240962 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.232832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:52797 (local address 127.12.110.193:44989)
0812 11:27:18.233043 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.233051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.233904 (+ 853us) server_negotiation.cc:407] Connection header received
0812 11:27:18.234102 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.234108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.234194 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.234297 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:18.235343 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.236087 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.238414 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.239919 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.240058 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.240168 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.240545 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.240695 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:27:18.240993 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.232821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:56099 (local address 127.12.110.195:38721)
0812 11:27:18.233043 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.233051 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.233904 (+ 853us) server_negotiation.cc:407] Connection header received
0812 11:27:18.234102 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.234108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.234190 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.234296 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:18.235298 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.236116 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.238799 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.239965 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.240187 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.240259 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.240628 (+ 369us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.240767 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:27:18.241101 6005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.232532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44989 (local address 127.12.110.194:52797)
0812 11:27:18.233700 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:18.233725 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:18.233920 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:18.234353 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:18.234360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:18.234409 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:18.235073 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.235084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.236245 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.236251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.238159 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.238173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.240176 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.240183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.240363 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.240392 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:27:18.240474 (+ 82us) client_negotiation.cc:240] Negotiation successful
0812 11:27:18.240617 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"mutex_wait_us":49,"spinlock_wait_cycles":11264,"thread_start_us":717,"threads_started":1}
I20260812 11:27:18.241194 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.232607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38721 (local address 127.12.110.194:56099)
0812 11:27:18.233760 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:18.233777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:18.233955 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:18.234353 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:18.234360 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:18.234411 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:18.235055 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.235073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.236292 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.236296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.238522 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:18.238573 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.240265 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.240276 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:18.240427 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.240476 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 11:27:18.240581 (+ 105us) client_negotiation.cc:240] Negotiation successful
0812 11:27:18.240720 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"mutex_wait_us":1,"spinlock_wait_cycles":44800,"thread_start_us":713,"threads_started":1}
I20260812 11:27:18.242303 5872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136f3dc84c7b49eb9d7889405d1f04ac" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03d318167eb45aaad2372df724a033f" is_pre_election: true
I20260812 11:27:18.242331 5376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136f3dc84c7b49eb9d7889405d1f04ac" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3134433d7489452ca30c4e706af4bc34" is_pre_election: true
I20260812 11:27:18.242926 5872 raft_consensus.cc:2454] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 0.
I20260812 11:27:18.242959 5376 raft_consensus.cc:2454] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 0.
I20260812 11:27:18.243790 5501 leader_election.cc:302] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3134433d7489452ca30c4e706af4bc34, c03d318167eb45aaad2372df724a033f, c740c3ffa1664c3d86e9d74840581d1a; no voters:
I20260812 11:27:18.244458 6004 raft_consensus.cc:2781] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:27:18.244694 6004 raft_consensus.cc:482] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:27:18.244867 6004 raft_consensus.cc:3037] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:18.248157 6004 raft_consensus.cc:504] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
I20260812 11:27:18.249496 6004 leader_election.cc:288] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 election: Requested vote from peers 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989), c03d318167eb45aaad2372df724a033f (127.12.110.195:38721)
I20260812 11:27:18.249945 5376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136f3dc84c7b49eb9d7889405d1f04ac" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3134433d7489452ca30c4e706af4bc34"
I20260812 11:27:18.250181 5376 raft_consensus.cc:3037] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:18.250095 5872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "136f3dc84c7b49eb9d7889405d1f04ac" candidate_uuid: "c740c3ffa1664c3d86e9d74840581d1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03d318167eb45aaad2372df724a033f"
I20260812 11:27:18.250365 5872 raft_consensus.cc:3037] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:18.253357 5376 raft_consensus.cc:2454] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 1.
I20260812 11:27:18.253451 5872 raft_consensus.cc:2454] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c740c3ffa1664c3d86e9d74840581d1a in term 1.
I20260812 11:27:18.254030 5497 leader_election.cc:302] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3134433d7489452ca30c4e706af4bc34, c740c3ffa1664c3d86e9d74840581d1a; no voters:
I20260812 11:27:18.254415 6004 raft_consensus.cc:2781] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:27:18.255297 6004 raft_consensus.cc:686] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 1 LEADER]: Becoming Leader. State: Replica: c740c3ffa1664c3d86e9d74840581d1a, State: Running, Role: LEADER
I20260812 11:27:18.255846 6004 consensus_queue.cc:237] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } }
W20260812 11:27:18.258599 5958 tablet.cc:2369] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:27:18.261125 5129 catalog_manager.cc:5495] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a reported cstate change: term changed from 0 to 1, leader changed from <none> to c740c3ffa1664c3d86e9d74840581d1a (127.12.110.194). New cstate: current_term: 1 leader_uuid: "c740c3ffa1664c3d86e9d74840581d1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c740c3ffa1664c3d86e9d74840581d1a" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 33759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 } health_report { overall_health: UNKNOWN } } }
W20260812 11:27:18.273555 5703 tablet.cc:2369] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:27:18.283843 5460 tablet.cc:2369] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:27:18.323210 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.317578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.12.110.194:33759)
0812 11:27:18.317900 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.317917 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.317945 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:27:18.318061 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.318066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.318157 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.318326 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:27:18.318793 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.319531 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.320956 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.321352 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.321492 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.322003 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:27:18.322509 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:27:18.322722 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.322830 (+ 108us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.322932 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260812 11:27:18.332703 5872 raft_consensus.cc:1261] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Refusing update from remote peer c740c3ffa1664c3d86e9d74840581d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:18.332703 5376 raft_consensus.cc:1261] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Refusing update from remote peer c740c3ffa1664c3d86e9d74840581d1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:18.334126 6010 consensus_queue.cc:1035] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03d318167eb45aaad2372df724a033f" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:18.334653 6004 consensus_queue.cc:1035] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3134433d7489452ca30c4e706af4bc34" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:18.365100 5129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:18.365345 5129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:18.370087 5129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Sending DeleteTablet for 3 replicas of tablet 136f3dc84c7b49eb9d7889405d1f04ac
I20260812 11:27:18.371454 5583 tablet_service.cc:1514] Processing DeleteTablet for tablet 136f3dc84c7b49eb9d7889405d1f04ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:18 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:50360
I20260812 11:27:18.371537 5343 tablet_service.cc:1514] Processing DeleteTablet for tablet 136f3dc84c7b49eb9d7889405d1f04ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:18 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:53712
I20260812 11:27:18.371595 5838 tablet_service.cc:1514] Processing DeleteTablet for tablet 136f3dc84c7b49eb9d7889405d1f04ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:18 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:51290
I20260812 11:27:18.372359 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 5206
I20260812 11:27:18.373616 6039 tablet_replica.cc:330] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: stopping tablet replica
I20260812 11:27:18.373806 6042 tablet_replica.cc:330] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: stopping tablet replica
I20260812 11:27:18.374073 6039 raft_consensus.cc:2229] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:27:18.374157 6042 raft_consensus.cc:2229] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:18.374732 6042 raft_consensus.cc:2258] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:18.374753 6039 raft_consensus.cc:2258] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:18.375533 6040 tablet_replica.cc:330] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: stopping tablet replica
I20260812 11:27:18.375926 6040 raft_consensus.cc:2229] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:18.376420 6040 raft_consensus.cc:2258] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:18.377063 6039 ts_tablet_manager.cc:1905] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:18.378398 6040 ts_tablet_manager.cc:1905] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:18.379424 6042 ts_tablet_manager.cc:1905] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:18.384630 6039 ts_tablet_manager.cc:1918] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:18.384793 6039 log.cc:1196] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/136f3dc84c7b49eb9d7889405d1f04ac
I20260812 11:27:18.385337 6039 ts_tablet_manager.cc:1939] T 136f3dc84c7b49eb9d7889405d1f04ac P c740c3ffa1664c3d86e9d74840581d1a: Deleting consensus metadata
I20260812 11:27:18.385725 6040 ts_tablet_manager.cc:1918] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:18.385890 6040 log.cc:1196] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/136f3dc84c7b49eb9d7889405d1f04ac
I20260812 11:27:18.386449 6040 ts_tablet_manager.cc:1939] T 136f3dc84c7b49eb9d7889405d1f04ac P c03d318167eb45aaad2372df724a033f: Deleting consensus metadata
I20260812 11:27:18.387418 5105 catalog_manager.cc:4841] TS c740c3ffa1664c3d86e9d74840581d1a (127.12.110.194:33759): tablet 136f3dc84c7b49eb9d7889405d1f04ac (table test-table [id=ab195f46bdaf442f8be7e76d13637482]) successfully deleted
I20260812 11:27:18.388207 5108 catalog_manager.cc:4841] TS c03d318167eb45aaad2372df724a033f (127.12.110.195:38721): tablet 136f3dc84c7b49eb9d7889405d1f04ac (table test-table [id=ab195f46bdaf442f8be7e76d13637482]) successfully deleted
I20260812 11:27:18.390775 5979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.374362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.12.110.193:44989)
0812 11:27:18.374643 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.374652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.374678 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:27:18.374755 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.374764 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.374865 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.374994 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:18.376682 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.377718 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.379821 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.380261 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.380399 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.383966 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:18.383978 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:18.383991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:18.384026 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:18.387399 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.388402 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.388413 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.388418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.388590 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.388999 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.389006 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.389009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.389400 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:18.389737 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.390070 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.390334 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:27:18.391501 5447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:18.490219 6042 ts_tablet_manager.cc:1918] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:18.490438 6042 log.cc:1196] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/136f3dc84c7b49eb9d7889405d1f04ac
I20260812 11:27:18.491364 6042 ts_tablet_manager.cc:1939] T 136f3dc84c7b49eb9d7889405d1f04ac P 3134433d7489452ca30c4e706af4bc34: Deleting consensus metadata
I20260812 11:27:18.493674 5106 catalog_manager.cc:4841] TS 3134433d7489452ca30c4e706af4bc34 (127.12.110.193:44989): tablet 136f3dc84c7b49eb9d7889405d1f04ac (table test-table [id=ab195f46bdaf442f8be7e76d13637482]) successfully deleted
I20260812 11:27:18.493836 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 5206
I20260812 11:27:18.533051 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 5466
I20260812 11:27:18.551373 5973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.533823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50376 (local address 127.12.110.194:33759)
0812 11:27:18.534007 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.534014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.534130 (+ 116us) server_negotiation.cc:407] Connection header received
0812 11:27:18.534341 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.534350 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.534428 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.534581 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:18.536503 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.537738 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.539902 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.540612 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.540736 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.544310 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:18.544322 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:18.544335 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:18.544375 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:18.547745 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.548622 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.548629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.548632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.548838 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.549687 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.549691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.549694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.550114 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:18.550448 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.550912 (+ 464us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.551097 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:27:18.552201 5694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:18.668427 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 5466
I20260812 11:27:18.700882 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 5708
I20260812 11:27:18.718273 5982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:18.701699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.12.110.195:38721)
0812 11:27:18.701941 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:18.701948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:18.702079 (+ 131us) server_negotiation.cc:407] Connection header received
0812 11:27:18.702357 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:18.702362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:18.702433 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:18.702580 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:18.705019 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.705931 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.708147 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:18.708612 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:18.708767 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:18.711780 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:18.711893 (+ 113us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:18.711912 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:18.711958 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:18.715055 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.715878 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.715886 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.715888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.716052 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:18.716502 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:18.716508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:18.716510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:18.716908 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:18.717241 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:18.717625 (+ 384us) server_negotiation.cc:299] Negotiation successful
0812 11:27:18.717869 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 11:27:18.719149 5949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:18.815261 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 5708
I20260812 11:27:18.839905 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 5075
I20260812 11:27:18.841352 5184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:18.957965 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 5075
I20260812 11:27:18.991215 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37049
--webserver_interface=127.12.110.254
--webserver_port=43215
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:19.363730 6052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:19.363998 6052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:19.364090 6052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:19.371610 6052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:19.371728 6052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:19.371775 6052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:19.371835 6052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:19.371881 6052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:19.382489 6052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=43215
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:19.383949 6052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:19.385322 6052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:19.398469 6067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:19.398630 6063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:19.398754 6065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:19.399823 6052 server_base.cc:1034] running on GCE node
I20260812 11:27:19.400652 6052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:19.402077 6052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:19.403291 6052 hybrid_clock.cc:648] HybridClock initialized: now 1786534039403225 us; error 53 us; skew 500 ppm
I20260812 11:27:19.403685 6052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:19 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534039, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:19.406957 6052 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:27:19.408475 6052 webserver.cc:460] Webserver started at http://127.12.110.254:43215/ using document root <none> and password file <none>
I20260812 11:27:19.409192 6052 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:19.409348 6052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:19.414798 6052 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:27:19.417685 6083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:19.419384 6052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:27:19.419589 6052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "c8376d5b8434490e9e9715aef2d44306"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "f29e63b4bc95c586b8dd1484c3476ed9"
server_key_iv: "957b5dd3186bed7cb53b6763c4762adc"
server_key_version: "encryptionkey@0"
I20260812 11:27:19.420639 6052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:19.437956 6052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:19.443683 6052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:19.444013 6052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:19.466485 6052 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37049
I20260812 11:27:19.466445 6181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37049 every 8 connection(s)
I20260812 11:27:19.469405 6052 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:27:19.475461 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 6052
I20260812 11:27:19.476373 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:44989
--local_ip_for_outbound_sockets=127.12.110.193
--tserver_master_addrs=127.12.110.254:37049
--webserver_port=45111
--webserver_interface=127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:27:19.478127 6182 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:27:19.481796 6182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Bootstrap starting.
I20260812 11:27:19.504755 6182 log.cc:824] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:19.522455 6182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:27:19.523164 6182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306: Bootstrap complete.
I20260812 11:27:19.533833 6182 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:19.534932 6182 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8376d5b8434490e9e9715aef2d44306, State: Initialized, Role: FOLLOWER
I20260812 11:27:19.535676 6182 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:19.535962 6182 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:27:19.536095 6182 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:27:19.536305 6182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:27:19.540750 6182 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:19.541314 6182 leader_election.cc:302] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8376d5b8434490e9e9715aef2d44306; no voters:
I20260812 11:27:19.542068 6182 leader_election.cc:288] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 11:27:19.542233 6190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 3 FOLLOWER]: Leader election won for term 3
I20260812 11:27:19.543601 6190 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [term 3 LEADER]: Becoming Leader. State: Replica: c8376d5b8434490e9e9715aef2d44306, State: Running, Role: LEADER
I20260812 11:27:19.544229 6190 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } }
I20260812 11:27:19.544554 6182 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:27:19.547171 6191 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c8376d5b8434490e9e9715aef2d44306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } } }
I20260812 11:27:19.547652 6191 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:19.551582 6192 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8376d5b8434490e9e9715aef2d44306. Latest consensus state: current_term: 3 leader_uuid: "c8376d5b8434490e9e9715aef2d44306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8376d5b8434490e9e9715aef2d44306" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37049 } } }
I20260812 11:27:19.552012 6192 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8376d5b8434490e9e9715aef2d44306 [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:19.553613 6199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:27:19.562214 6199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:27:19.563724 6199 catalog_manager.cc:1259] Loaded cluster ID: 6bba7cf2a4394ae89b13ddefed10374f
I20260812 11:27:19.563822 6199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260812 11:27:19.566134 6199 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f08a60f79ad google::LogMessage::Fail() at ??:0
@ 0x7f08a60fa88a google::LogMessage::SendToLog() at ??:0
@ 0x7f08a60f74de google::LogMessage::Flush() at ??:0
@ 0x7f08a60f8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f08bbb2a512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f08bbb27976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f08bbbaf610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f08bbbaf439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f08a8d43345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f08a8d4691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f08a8d46799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f08a8d21eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f08abffb6db start_thread at ??:0
@ 0x7f08a4f9161f clone at ??:0
W20260812 11:27:19.887794 6186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:19.888144 6186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:19.888332 6186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:19.895529 6186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:19.895684 6186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:19.895857 6186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:27:19.905630 6186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:44989
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=45111
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:19.907044 6186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:19.908413 6186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:19.922144 6250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:19.922315 6251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:19.922518 6254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:19.924203 6186 server_base.cc:1034] running on GCE node
I20260812 11:27:19.925076 6186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:19.926095 6186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:19.927361 6186 hybrid_clock.cc:648] HybridClock initialized: now 1786534039927322 us; error 64 us; skew 500 ppm
I20260812 11:27:19.927726 6186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:19 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534039, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:19.931250 6186 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:27:19.932809 6186 webserver.cc:460] Webserver started at http://127.12.110.193:45111/ using document root <none> and password file <none>
I20260812 11:27:19.933555 6186 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:19.933714 6186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:19.939502 6186 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:27:19.942363 6264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:19.943776 6186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:27:19.944006 6186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "3134433d7489452ca30c4e706af4bc34"
format_stamp: "Formatted at 2026-08-12 11:27:13 on dist-test-slave-h6n0"
server_key: "8b96abe3ad7c297bb0c5ad4cba636d46"
server_key_iv: "cce558edbe5bd0f1ebe3d12c642860ff"
server_key_version: "encryptionkey@0"
I20260812 11:27:19.945078 6186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:19.972441 6186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:19.976846 6186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:19.977154 6186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:19.978186 6186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:19.979588 6186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:19.979717 6186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:19.979853 6186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:19.979956 6186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:20.016997 6186 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:44989
I20260812 11:27:20.017058 6470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:44989 every 8 connection(s)
I20260812 11:27:20.018492 6186 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:27:20.021257 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 6186
I20260812 11:27:20.021932 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:33759
--local_ip_for_outbound_sockets=127.12.110.194
--tserver_master_addrs=127.12.110.254:37049
--webserver_port=44297
--webserver_interface=127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:27:20.024338 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:20.022138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:43921)
0812 11:27:20.023116 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:20.023459 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":413,"threads_started":1}
W20260812 11:27:20.025110 6471 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:37049 (0 consecutive failures): Network error: Failed to ping master at 127.12.110.254:37049: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
I20260812 11:27:20.026114 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:20.025759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:43297)
0812 11:27:20.025924 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:20.025999 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 11:27:20.027094 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:20.026732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:33771)
0812 11:27:20.026885 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:20.026979 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:27:20.027374 6471 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:27:20.380777 6477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:20.381072 6477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:20.381242 6477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:20.388553 6477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:20.388665 6477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:20.388854 6477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:27:20.400177 6477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:33759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=44297
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:20.402176 6477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:20.403586 6477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:20.417728 6494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:20.417760 6487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:20.417886 6488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:20.419418 6477 server_base.cc:1034] running on GCE node
I20260812 11:27:20.420364 6477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:20.421854 6477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:20.423159 6477 hybrid_clock.cc:648] HybridClock initialized: now 1786534040423087 us; error 77 us; skew 500 ppm
I20260812 11:27:20.423537 6477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:20 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534040, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:20.427515 6477 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:27:20.430001 6477 webserver.cc:460] Webserver started at http://127.12.110.194:44297/ using document root <none> and password file <none>
I20260812 11:27:20.430871 6477 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:20.431037 6477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:20.436828 6477 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:27:20.439751 6508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:20.441256 6477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:27:20.441459 6477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "c740c3ffa1664c3d86e9d74840581d1a"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "d1cac2cdcd6bc6d817e6acb638dd3d8f"
server_key_iv: "4bac9a2177e40fe418ee424347690a00"
server_key_version: "encryptionkey@0"
I20260812 11:27:20.442667 6477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:20.469484 6477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:20.474875 6477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:20.475186 6477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:20.476307 6477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:20.477772 6477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:20.477919 6477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:20.478053 6477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:20.478152 6477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:20.516122 6477 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:33759
I20260812 11:27:20.516426 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:33759 every 8 connection(s)
I20260812 11:27:20.517658 6477 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:27:20.522468 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:20.520663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:57741)
0812 11:27:20.521402 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:20.521644 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":300,"threads_started":1}
W20260812 11:27:20.523185 6705 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:37049 (0 consecutive failures): Network error: Failed to ping master at 127.12.110.254:37049: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
I20260812 11:27:20.524012 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:20.523628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:55559)
0812 11:27:20.523836 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:20.523906 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 11:27:20.524893 6708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:20.524561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:53445)
0812 11:27:20.524691 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:20.524774 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 11:27:20.525149 6705 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:27:20.525493 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 6477
I20260812 11:27:20.526190 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:38721
--local_ip_for_outbound_sockets=127.12.110.195
--tserver_master_addrs=127.12.110.254:37049
--webserver_port=42481
--webserver_interface=127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:20.871078 6710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:20.871405 6710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:20.871606 6710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:20.878962 6710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:20.879142 6710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:20.879333 6710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:27:20.889050 6710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:38721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=42481
--tserver_master_addrs=127.12.110.254:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:20.890982 6710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:20.892882 6710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:20.907804 6717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:20.907830 6718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:20.907756 6720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:20.908981 6710 server_base.cc:1034] running on GCE node
I20260812 11:27:20.909575 6710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:20.910447 6710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:20.911649 6710 hybrid_clock.cc:648] HybridClock initialized: now 1786534040911605 us; error 30 us; skew 500 ppm
I20260812 11:27:20.911933 6710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:20 dist-test-slave-h6n0 krb5kdc[4095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534040, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:20.919999 6710 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:27:20.921562 6710 webserver.cc:460] Webserver started at http://127.12.110.195:42481/ using document root <none> and password file <none>
I20260812 11:27:20.922204 6710 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:20.922312 6710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:20.927856 6710 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:27:20.930611 6734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:20.932152 6710 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:27:20.932293 6710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "c03d318167eb45aaad2372df724a033f"
format_stamp: "Formatted at 2026-08-12 11:27:14 on dist-test-slave-h6n0"
server_key: "e7c05aa963d71a9a686d3aaa59aa789d"
server_key_iv: "f5908f902a60c3983dd4fa6b86c6a018"
server_key_version: "encryptionkey@0"
I20260812 11:27:20.933251 6710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:20.947376 6710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:20.951946 6710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:20.952171 6710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:20.953083 6710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:20.954366 6710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:20.954466 6710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:20.954556 6710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:20.954615 6710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 11:27:20.991232 6710 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:38721
I20260812 11:27:20.991328 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:38721 every 8 connection(s)
I20260812 11:27:20.992748 6710 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:27:20.998700 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 6710
I20260812 11:27:20.998733 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:20.996082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:35273)
0812 11:27:20.997136 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:20.997559 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":429,"threads_started":1}
W20260812 11:27:20.999497 6943 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:37049 (0 consecutive failures): Network error: Failed to ping master at 127.12.110.254:37049: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
I20260812 11:27:21.000765 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:21.000126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:35791)
0812 11:27:21.000421 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:21.000558 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260812 11:27:21.001873 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:21.001442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:44675)
0812 11:27:21.001621 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:21.001729 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260812 11:27:21.002170 6943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:27:21.029224 6949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:21.028095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:57497)
0812 11:27:21.028726 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:21.028876 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":370,"threads_started":1}
I20260812 11:27:21.526522 6950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:21.525665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:42533)
0812 11:27:21.526196 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:21.526309 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":298,"threads_started":1}
I20260812 11:27:22.003860 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:22.002881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:44691)
0812 11:27:22.003434 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:22.003572 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":310,"threads_started":1}
I20260812 11:27:22.031261 6952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:22.030448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:48143)
0812 11:27:22.031010 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:22.031092 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":338,"threads_started":1}
I20260812 11:27:22.528525 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:22.527652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:43907)
0812 11:27:22.528257 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:22.528352 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":291,"threads_started":1}
I20260812 11:27:23.005723 6954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:23.004867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:45571)
0812 11:27:23.005444 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:23.005546 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":265,"threads_started":1}
I20260812 11:27:23.032938 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:23.032245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:41789)
0812 11:27:23.032668 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:23.032751 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20260812 11:27:23.530709 6956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:23.529479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:51735)
0812 11:27:23.530233 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:23.530368 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":380,"threads_started":1}
I20260812 11:27:24.007632 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:24.006850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:44001)
0812 11:27:24.007382 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:24.007464 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":271,"threads_started":1}
I20260812 11:27:24.034914 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:24.034095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:37873)
0812 11:27:24.034626 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:24.034719 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":230,"threads_started":1}
I20260812 11:27:24.532500 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:24.531729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:57353)
0812 11:27:24.532222 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:24.532308 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":265,"threads_started":1}
I20260812 11:27:25.009374 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:25.008595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:56489)
0812 11:27:25.009093 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:25.009175 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":268,"threads_started":1}
I20260812 11:27:25.036834 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:25.035922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:37145)
0812 11:27:25.036401 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:25.036525 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
I20260812 11:27:25.534197 6968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:25.533449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:35721)
0812 11:27:25.533932 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:25.534027 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":273,"threads_started":1}
I20260812 11:27:26.011373 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:26.010471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:57661)
0812 11:27:26.011085 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:26.011173 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":358,"threads_started":1}
I20260812 11:27:26.038477 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:26.037732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:55529)
0812 11:27:26.038209 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:26.038297 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":272,"threads_started":1}
I20260812 11:27:26.536068 6976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:26.535196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:33617)
0812 11:27:26.535793 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:26.535880 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":319,"threads_started":1}
I20260812 11:27:27.013224 6977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:27.012335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:56867)
0812 11:27:27.012842 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:27.012942 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":275,"threads_started":1}
I20260812 11:27:27.040736 6978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:27.039678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:34317)
0812 11:27:27.040413 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:27.040510 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":231,"threads_started":1}
I20260812 11:27:27.537963 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:27.537039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:39465)
0812 11:27:27.537586 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:27.537730 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":214,"threads_started":1}
I20260812 11:27:28.015581 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:28.014251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:41385)
0812 11:27:28.014899 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:28.014991 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":346,"threads_started":1}
I20260812 11:27:28.042711 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:28.041813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:41489)
0812 11:27:28.042390 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:28.042475 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":323,"threads_started":1}
I20260812 11:27:28.540412 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:28.539448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:57497)
0812 11:27:28.540091 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:28.540208 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":310,"threads_started":1}
I20260812 11:27:29.018234 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:29.016953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:49285)
0812 11:27:29.017860 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:29.018018 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":487,"threads_started":1}
I20260812 11:27:29.044936 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:29.044034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:49073)
0812 11:27:29.044631 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:29.044723 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":278,"threads_started":1}
I20260812 11:27:29.542747 6991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:29.541537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:33725)
0812 11:27:29.542366 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:29.542456 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":617,"threads_started":1}
I20260812 11:27:30.020453 6992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:30.019373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:54379)
0812 11:27:30.020097 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:30.020230 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":359,"threads_started":1}
I20260812 11:27:30.047406 6993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:30.046474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:54447)
0812 11:27:30.047090 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:30.047235 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":372,"threads_started":1}
I20260812 11:27:30.545234 6994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:30.544239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:34429)
0812 11:27:30.544977 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:30.545066 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":465,"threads_started":1}
I20260812 11:27:31.023316 6995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:31.021607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:39325)
0812 11:27:31.022850 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:31.022986 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":762,"threads_started":1}
I20260812 11:27:31.049324 6996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:31.048425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:57145)
0812 11:27:31.048982 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:31.049093 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":287,"threads_started":1}
I20260812 11:27:31.547143 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:31.546234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:55959)
0812 11:27:31.546786 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:31.546906 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":230,"threads_started":1}
I20260812 11:27:32.025379 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:32.024570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:35759)
0812 11:27:32.025112 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:32.025203 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":311,"threads_started":1}
I20260812 11:27:32.051007 7002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:32.050226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:40355)
0812 11:27:32.050767 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:32.050847 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":328,"threads_started":1}
I20260812 11:27:32.549341 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:32.548427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:55945)
0812 11:27:32.549058 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:32.549157 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":407,"threads_started":1}
I20260812 11:27:33.027518 7005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:33.026340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:41881)
0812 11:27:33.027111 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:33.027343 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":425,"threads_started":1}
I20260812 11:27:33.052601 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:33.051801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:55087)
0812 11:27:33.052321 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:33.052418 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":309,"threads_started":1}
I20260812 11:27:33.551271 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:33.550397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:43261)
0812 11:27:33.550965 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:33.551083 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":332,"threads_started":1}
I20260812 11:27:34.029300 7011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:34.028515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:35769)
0812 11:27:34.029018 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:34.029115 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":240,"threads_started":1}
I20260812 11:27:34.054230 7015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:34.053449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:34297)
0812 11:27:34.053965 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:34.054045 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":282,"threads_started":1}
I20260812 11:27:34.553150 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:34.552306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:35537)
0812 11:27:34.552832 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:34.552917 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":236,"threads_started":1}
I20260812 11:27:35.031180 7018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:35.030245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:57555)
0812 11:27:35.030897 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:35.030987 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":226,"threads_started":1}
I20260812 11:27:35.056437 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:35.055504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.193:55013)
0812 11:27:35.056032 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:35.056208 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":269,"threads_started":1}
I20260812 11:27:35.555351 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:35.554510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.194:41241)
0812 11:27:35.555050 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:35.555167 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":229,"threads_started":1}
I20260812 11:27:36.001677 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 6186
I20260812 11:27:36.020586 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:36.003197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38260 (local address 127.12.110.193:44989)
0812 11:27:36.003872 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:36.003886 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:36.003927 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:27:36.005573 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:36.005583 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:36.005994 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:36.006295 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:36.007856 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:36.009086 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:36.010305 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:36.011065 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:36.011199 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:36.014063 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:36.014071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:36.014085 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:36.014135 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:36.016988 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:36.017649 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:36.017655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:36.017661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:36.017828 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:36.018266 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:36.018270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:36.018273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:36.018968 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:36.019271 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:36.019815 (+ 544us) server_negotiation.cc:299] Negotiation successful
0812 11:27:36.020279 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":343,"threads_started":1}
I20260812 11:27:36.022118 6466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:36.033437 7023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:36.032662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37049 (local address 127.12.110.195:55237)
0812 11:27:36.033153 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:36.033254 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:37049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":263,"threads_started":1}
I20260812 11:27:36.111552 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 6186
I20260812 11:27:36.148568 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 6477
I20260812 11:27:36.169958 7026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:36.149604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43630 (local address 127.12.110.194:33759)
0812 11:27:36.150189 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:36.150203 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:36.150240 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:27:36.151934 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:36.151949 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:36.152368 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:36.152935 (+ 567us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:36.155237 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:36.157020 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:36.158442 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:36.159217 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:36.159434 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:36.162264 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:36.162273 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:36.162289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:36.162332 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:36.166354 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:36.167181 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:36.167188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:36.167195 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:36.167389 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:36.167804 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:36.167809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:36.167814 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:36.168394 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:36.168776 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:36.169209 (+ 433us) server_negotiation.cc:299] Negotiation successful
0812 11:27:36.169651 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":277,"threads_started":1}
I20260812 11:27:36.171444 6699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:36.266716 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 6477
I20260812 11:27:36.296447 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 6710
I20260812 11:27:36.315979 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:36.297539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37292 (local address 127.12.110.195:38721)
0812 11:27:36.298107 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:36.298122 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:36.298167 (+ 45us) server_negotiation.cc:407] Connection header received
0812 11:27:36.299952 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:36.299962 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:36.300408 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:36.300720 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:36.302210 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:36.303530 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:36.305046 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:36.305927 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:36.306110 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:36.308899 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:36.308907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:36.308922 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:36.308971 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:36.311911 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:36.312601 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:36.312608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:36.312614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:36.312783 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:36.313537 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:36.313543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:36.313546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:36.314477 (+ 931us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:36.314840 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:36.315202 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 11:27:36.315671 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":251,"threads_started":1}
I20260812 11:27:36.318112 6932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:36.404863 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 6710
2026-08-12T11:27:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25369 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:27:36 dist-test-slave-h6n0 krb5kdc[7042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:27:36 dist-test-slave-h6n0 krb5kdc[7042](info): set up 2 sockets
Aug 12 11:27:36 dist-test-slave-h6n0 krb5kdc[7042](info): commencing operation
krb5kdc: starting...
W20260812 11:27:37.839708 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.384s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:27:38 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:27:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:27:38.179927 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42741
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:38.568274 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:38.568581 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:38.568761 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:38.576449 7063 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:27:38.576562 7063 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:27:38.576603 7063 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:27:38.576752 7063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:27:38.576859 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:38.576982 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:38.577049 7063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:27:38.577121 7063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:27:38.588872 7063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:38.590382 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:38.591907 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:38.606833 7076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:38.607210 7063 server_base.cc:1034] running on GCE node
W20260812 11:27:38.606913 7070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:38.606885 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:38.608927 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:38.611387 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:38.612725 7063 hybrid_clock.cc:648] HybridClock initialized: now 1786534058612632 us; error 77 us; skew 500 ppm
I20260812 11:27:38.613138 7063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:38 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534058, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:38.616648 7063 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:27:38.618337 7063 webserver.cc:460] Webserver started at http://127.12.110.254:34839/ using document root <none> and password file <none>
I20260812 11:27:38.619102 7063 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:38.619196 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:38.619467 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:38.622489 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "827b37303d194274b7bd989c63a22aab"
format_stamp: "Formatted at 2026-08-12 11:27:38 on dist-test-slave-h6n0"
server_key: "3db4854caaa4f53023fdc462a1d98ff2"
server_key_iv: "b57c5048ae13520a02f7fc57bb62cec2"
server_key_version: "encryptionkey@0"
I20260812 11:27:38.623308 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "827b37303d194274b7bd989c63a22aab"
format_stamp: "Formatted at 2026-08-12 11:27:38 on dist-test-slave-h6n0"
server_key: "3db4854caaa4f53023fdc462a1d98ff2"
server_key_iv: "b57c5048ae13520a02f7fc57bb62cec2"
server_key_version: "encryptionkey@0"
I20260812 11:27:38.629520 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260812 11:27:38.633307 7087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:38.635521 7063 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260812 11:27:38.635684 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "827b37303d194274b7bd989c63a22aab"
format_stamp: "Formatted at 2026-08-12 11:27:38 on dist-test-slave-h6n0"
server_key: "3db4854caaa4f53023fdc462a1d98ff2"
server_key_iv: "b57c5048ae13520a02f7fc57bb62cec2"
server_key_version: "encryptionkey@0"
I20260812 11:27:38.635882 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:38.655285 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:38.659979 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:38.660218 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:38.684337 7063 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:42741
I20260812 11:27:38.684362 7179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:42741 every 8 connection(s)
I20260812 11:27:38.685895 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:27:38.691032 7180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:38.696434 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 7063
I20260812 11:27:38.696691 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:27:38.698062 12731 external_mini_cluster.cc:1428] Setting key 179eaf66808edf1a09d7ee488bf3a5d8
I20260812 11:27:38.704720 7180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Bootstrap starting.
I20260812 11:27:38.707955 7180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:38.709020 7180 log.cc:824] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:38.712683 7180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: No bootstrap required, opened a new log
Aug 12 11:27:38 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:38.723690 7180 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:27:38.724618 7180 raft_consensus.cc:374] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:38.724754 7180 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827b37303d194274b7bd989c63a22aab, State: Initialized, Role: FOLLOWER
I20260812 11:27:38.725521 7180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:27:38.725823 7180 raft_consensus.cc:388] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:27:38.725992 7180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:27:38.726148 7186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:38.700751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59550 (local address 127.12.110.254:42741)
0812 11:27:38.701834 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:38.701849 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:38.701893 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:27:38.703771 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:38.703795 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:38.704168 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:38.704570 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:38.706205 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:38.707580 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:38.709148 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:38.710189 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:38.710407 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:38.715745 (+ 5338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:38.715756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:38.715775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:38.715835 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:38.720119 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:38.720975 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:38.720985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:38.721000 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:38.721232 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:38.722019 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:38.722026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:38.722031 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:38.722804 (+ 773us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:38.723438 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:38.723935 (+ 497us) server_negotiation.cc:299] Negotiation successful
0812 11:27:38.724668 (+ 733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":868,"thread_start_us":502,"threads_started":1}
I20260812 11:27:38.726208 7180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:38.729415 7180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:27:38.729956 7180 leader_election.cc:302] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 827b37303d194274b7bd989c63a22aab; no voters:
I20260812 11:27:38.730655 7180 leader_election.cc:288] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:27:38.730820 7189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:27:38.732234 7189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 LEADER]: Becoming Leader. State: Replica: 827b37303d194274b7bd989c63a22aab, State: Running, Role: LEADER
I20260812 11:27:38.732918 7189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:27:38.733172 7180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:27:38.736622 7194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827b37303d194274b7bd989c63a22aab. Latest consensus state: current_term: 1 leader_uuid: "827b37303d194274b7bd989c63a22aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } } }
I20260812 11:27:38.736272 7191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "827b37303d194274b7bd989c63a22aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } } }
I20260812 11:27:38.737236 7194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:38.737257 7191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: This master's current role is: LEADER
I20260812 11:27:38.738927 7204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:27:38.744939 7204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:27:38.757855 7204 catalog_manager.cc:1347] Generated new cluster ID: 58e328ac8b9843928778e12c03a50051
I20260812 11:27:38.758018 7204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:27:38.782712 7204 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:27:38.784130 7204 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:27:38.790457 7204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Generated new TSK 0
I20260812 11:27:38.791363 7204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:27:38.884924 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42741
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:39.288944 7230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:39.289439 7230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:39.289608 7230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:39.297770 7230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:39.297879 7230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:39.298089 7230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:27:39.310138 7230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:39.311632 7230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:39.312970 7230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:39.326474 7240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:39.326361 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:39.326555 7244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:39.327706 7230 server_base.cc:1034] running on GCE node
I20260812 11:27:39.328293 7230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:39.329216 7230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:39.330436 7230 hybrid_clock.cc:648] HybridClock initialized: now 1786534059330380 us; error 56 us; skew 500 ppm
I20260812 11:27:39.330789 7230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:39 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534059, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:39.337980 7230 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:27:39.339689 7230 webserver.cc:460] Webserver started at http://127.12.110.193:46107/ using document root <none> and password file <none>
I20260812 11:27:39.340405 7230 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:39.340515 7230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:39.340835 7230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:39.343871 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "5ee8024bd1f348b27864dd16e35b9f26"
server_key_iv: "7238f6866241412ba3a98edb1305b66a"
server_key_version: "encryptionkey@0"
I20260812 11:27:39.344610 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "5ee8024bd1f348b27864dd16e35b9f26"
server_key_iv: "7238f6866241412ba3a98edb1305b66a"
server_key_version: "encryptionkey@0"
I20260812 11:27:39.350440 7230 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260812 11:27:39.354123 7262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:39.356438 7230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260812 11:27:39.356603 7230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "5ee8024bd1f348b27864dd16e35b9f26"
server_key_iv: "7238f6866241412ba3a98edb1305b66a"
server_key_version: "encryptionkey@0"
I20260812 11:27:39.356808 7230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:39.377928 7230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:39.382490 7230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:39.382946 7230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:39.384043 7230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:39.385536 7230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:39.385614 7230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:39.385706 7230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:39.385752 7230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:39.427394 7230 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:37869
I20260812 11:27:39.427529 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:37869 every 8 connection(s)
I20260812 11:27:39.428759 7230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:27:39.437907 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 7230
I20260812 11:27:39.438283 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:27:39.439146 12731 external_mini_cluster.cc:1428] Setting key 74c22861fbd96298524ef73cc971b50c
Aug 12 11:27:39 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534059, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:39.453179 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:39.431424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:56999 (local address 127.12.110.254:42741)
0812 11:27:39.431897 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:39.431906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:39.432878 (+ 972us) server_negotiation.cc:407] Connection header received
0812 11:27:39.435035 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:39.435040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:39.435110 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:39.435208 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:39.436790 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:39.437807 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:39.440108 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:39.440538 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:39.440647 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:39.445157 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:39.445167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:39.445176 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:39.445227 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:39.449398 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:39.450247 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:39.450257 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:39.450264 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:39.450433 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:39.451004 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:39.451013 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:39.451020 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:39.451445 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:39.451618 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:39.452351 (+ 733us) server_negotiation.cc:299] Negotiation successful
0812 11:27:39.452645 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260812 11:27:39.453811 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:39.431963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:56999)
0812 11:27:39.432679 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:39.432717 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:39.434779 (+ 2062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:39.435406 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:39.435413 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:39.435973 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:39.436590 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:39.436610 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:39.437997 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:39.438003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:39.439916 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:39.439958 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:39.440818 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:39.440827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:39.441139 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:39.441923 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:39.441966 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:39.444910 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:39.449592 (+ 4682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:39.449600 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:39.449613 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:39.450092 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:39.450637 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:39.450641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:39.450643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:39.450855 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:39.451594 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:39.451600 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:39.451931 (+ 331us) client_negotiation.cc:769] Sending connection context
0812 11:27:39.452170 (+ 239us) client_negotiation.cc:240] Negotiation successful
0812 11:27:39.452586 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":277,"threads_started":1}
I20260812 11:27:39.455504 7454 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42741
I20260812 11:27:39.455843 7454 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:39.456668 7454 heartbeater.cc:511] Master 127.12.110.254:42741 requested a full tablet report, sending...
I20260812 11:27:39.459260 7114 ts_manager.cc:194] Registered new tserver with Master: c6523020d0bf4ddfbe9ad4c3befcb86d (127.12.110.193:37869)
I20260812 11:27:39.460848 7114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:56999
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:27:39.522002 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42741
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:27:39.887447 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:39.887815 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:39.888000 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:39.897598 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:39.897843 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:39.898073 7466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:27:39.910017 7466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:39.911490 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:39.913081 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:39.928849 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:39.928980 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:39.928923 7478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:39.930188 7466 server_base.cc:1034] running on GCE node
I20260812 11:27:39.930927 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:39.931960 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:39.933231 7466 hybrid_clock.cc:648] HybridClock initialized: now 1786534059933164 us; error 50 us; skew 500 ppm
I20260812 11:27:39.933599 7466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:39 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534059, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:39.937997 7466 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:27:39.940497 7466 webserver.cc:460] Webserver started at http://127.12.110.194:40647/ using document root <none> and password file <none>
I20260812 11:27:39.941273 7466 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:39.941442 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:39.941807 7466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:39.945294 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "42b3d5d1573446f683cf83e49f8d85d4"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "490dc55451cb64a89310146c9def3872"
server_key_iv: "453b01f23afbf66db66cf4f7d38e65a3"
server_key_version: "encryptionkey@0"
I20260812 11:27:39.946182 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "42b3d5d1573446f683cf83e49f8d85d4"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "490dc55451cb64a89310146c9def3872"
server_key_iv: "453b01f23afbf66db66cf4f7d38e65a3"
server_key_version: "encryptionkey@0"
I20260812 11:27:39.952006 7466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260812 11:27:39.955865 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:39.957571 7466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:27:39.957754 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "42b3d5d1573446f683cf83e49f8d85d4"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "490dc55451cb64a89310146c9def3872"
server_key_iv: "453b01f23afbf66db66cf4f7d38e65a3"
server_key_version: "encryptionkey@0"
I20260812 11:27:39.957975 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:39.985798 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:39.990677 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:39.990991 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:39.992002 7466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:39.993464 7466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:39.993677 7466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:39.993846 7466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:39.993924 7466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:40.035511 7466 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:45647
I20260812 11:27:40.035604 7694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:45647 every 8 connection(s)
I20260812 11:27:40.036937 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:27:40.039012 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 7466
I20260812 11:27:40.039248 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:27:40.039819 12731 external_mini_cluster.cc:1428] Setting key 6327ef7e7be14e82b93a3e46b7c51258
Aug 12 11:27:40 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534059, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:40.068488 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.040623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:39221 (local address 127.12.110.254:42741)
0812 11:27:40.047047 (+ 6424us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.047057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.047086 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:27:40.047426 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.047435 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.047532 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.047686 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:40.049467 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.050790 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.053372 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.054219 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.054395 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.060291 (+ 5896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:40.060297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:40.060300 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:40.060337 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:40.064120 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.065249 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.065254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.065257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.065427 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.066082 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.066087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.066090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.066633 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:40.066822 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.067919 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.068180 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6257,"thread_start_us":6176,"threads_started":1}
I20260812 11:27:40.069974 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.041214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:39221)
0812 11:27:40.042685 (+ 1471us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:40.042751 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:40.046815 (+ 4064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:40.047991 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:40.048002 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:40.048638 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:40.049259 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.049278 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.050998 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.051003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.052957 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.052984 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.054578 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.054589 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.055003 (+ 414us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.056200 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:40.056235 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:40.060001 (+ 3766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:40.064337 (+ 4336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:40.064348 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:40.064369 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:40.065086 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:40.065569 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:40.065576 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:40.065583 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:40.065874 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:40.066812 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:40.066823 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:40.067393 (+ 570us) client_negotiation.cc:769] Sending connection context
0812 11:27:40.067736 (+ 343us) client_negotiation.cc:240] Negotiation successful
0812 11:27:40.068381 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":518,"threads_started":1}
I20260812 11:27:40.071792 7696 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42741
I20260812 11:27:40.072157 7696 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:40.073117 7696 heartbeater.cc:511] Master 127.12.110.254:42741 requested a full tablet report, sending...
I20260812 11:27:40.074884 7114 ts_manager.cc:194] Registered new tserver with Master: 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
I20260812 11:27:40.075791 7114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:39221
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:27:40.121944 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42741
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:27:40.464684 7454 heartbeater.cc:503] Master 127.12.110.254:42741 was elected leader, sending a full tablet report...
W20260812 11:27:40.505007 7704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:27:40.505296 7704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:27:40.505393 7704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:27:40.512907 7704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:27:40.513023 7704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:27:40.513137 7704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:27:40.524811 7704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:27:40.526083 7704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:27:40.527465 7704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:27:40.543080 7713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:40.543219 7714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:27:40.542997 7716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:27:40.544049 7704 server_base.cc:1034] running on GCE node
I20260812 11:27:40.544705 7704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:27:40.545725 7704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:27:40.547025 7704 hybrid_clock.cc:648] HybridClock initialized: now 1786534060546949 us; error 29 us; skew 500 ppm
I20260812 11:27:40.547708 7704 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:27:40 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:27:40.551841 7704 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:27:40.553649 7704 webserver.cc:460] Webserver started at http://127.12.110.195:42089/ using document root <none> and password file <none>
I20260812 11:27:40.554468 7704 fs_manager.cc:359] Metadata directory not provided
I20260812 11:27:40.554693 7704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:27:40.555048 7704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:27:40.558323 7704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "a9f015c5d9844f7c82d84ccefa26e9a9"
format_stamp: "Formatted at 2026-08-12 11:27:40 on dist-test-slave-h6n0"
server_key: "3a5cbf03cf6126ae59b0a5f434aaeba9"
server_key_iv: "782c4b9d7f4c7aa28eb92f49ba6ac1f7"
server_key_version: "encryptionkey@0"
I20260812 11:27:40.559243 7704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "a9f015c5d9844f7c82d84ccefa26e9a9"
format_stamp: "Formatted at 2026-08-12 11:27:40 on dist-test-slave-h6n0"
server_key: "3a5cbf03cf6126ae59b0a5f434aaeba9"
server_key_iv: "782c4b9d7f4c7aa28eb92f49ba6ac1f7"
server_key_version: "encryptionkey@0"
I20260812 11:27:40.564976 7704 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:27:40.568843 7729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:40.570262 7704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:27:40.570482 7704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "a9f015c5d9844f7c82d84ccefa26e9a9"
format_stamp: "Formatted at 2026-08-12 11:27:40 on dist-test-slave-h6n0"
server_key: "3a5cbf03cf6126ae59b0a5f434aaeba9"
server_key_iv: "782c4b9d7f4c7aa28eb92f49ba6ac1f7"
server_key_version: "encryptionkey@0"
I20260812 11:27:40.570755 7704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:27:40.592880 7704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:27:40.597570 7704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:27:40.597926 7704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:27:40.599004 7704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:27:40.600384 7704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:27:40.600497 7704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:40.600660 7704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:27:40.600739 7704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:27:40.637261 7704 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:38413
I20260812 11:27:40.637499 7926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:38413 every 8 connection(s)
I20260812 11:27:40.638706 7704 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:27:40.639778 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 7704
I20260812 11:27:40.639988 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:27:40.640626 12731 external_mini_cluster.cc:1428] Setting key 10769529e54b0c84739a8fde1e80c183
Aug 12 11:27:40 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534060, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:27:40.662701 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.642092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:40811 (local address 127.12.110.254:42741)
0812 11:27:40.642865 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.642873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.643720 (+ 847us) server_negotiation.cc:407] Connection header received
0812 11:27:40.646224 (+ 2504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.646230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.646331 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.646465 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:40.648138 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.648970 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.650659 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.651090 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.651263 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.655078 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:40.655084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:40.655087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:40.655120 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:40.659024 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.659921 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.659926 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.659929 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.660064 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.660582 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.660587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.660589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.660949 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:40.661141 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.661939 (+ 798us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.662210 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":325,"threads_started":1}
I20260812 11:27:40.663653 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.642516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:40811)
0812 11:27:40.643543 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:40.643594 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:40.645875 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:40.646794 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:40.646801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:40.647357 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:27:40.647927 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.647945 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.649141 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.649148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.650448 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.650463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.651339 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.651349 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.651624 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.652355 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:27:40.652380 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:27:40.654836 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:27:40.659181 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:40.659191 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:40.659212 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:40.659761 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:40.660192 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:27:40.660197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:27:40.660201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:27:40.660437 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:27:40.661096 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:27:40.661105 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:27:40.661476 (+ 371us) client_negotiation.cc:769] Sending connection context
0812 11:27:40.661760 (+ 284us) client_negotiation.cc:240] Negotiation successful
0812 11:27:40.662334 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":391,"threads_started":1}
I20260812 11:27:40.665046 7927 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42741
I20260812 11:27:40.665339 7927 heartbeater.cc:464] Registering TS with master...
I20260812 11:27:40.666059 7927 heartbeater.cc:511] Master 127.12.110.254:42741 requested a full tablet report, sending...
I20260812 11:27:40.667524 7114 ts_manager.cc:194] Registered new tserver with Master: a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413)
I20260812 11:27:40.668391 7114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:40811
I20260812 11:27:40.670660 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:27:40.690757 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.675751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.12.110.254:42741)
0812 11:27:40.675924 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.675928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.676402 (+ 474us) server_negotiation.cc:407] Connection header received
0812 11:27:40.676504 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.676508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.676576 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.676657 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:40.678128 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.678967 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.680741 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.681181 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.681293 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.684666 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:40.684673 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:40.684676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:40.684711 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:40.687824 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.688570 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.688576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.688580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.688710 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.689108 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.689114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.689117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.689539 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:40.689843 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.690144 (+ 301us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.690405 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:27:40.695854 7114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:27:40.703173 7114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:27:40.743621 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.719174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60554 (local address 127.12.110.195:38413)
0812 11:27:40.732918 (+ 13744us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.732924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.732955 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:27:40.733039 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.733045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.733252 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.733491 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:40.734358 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.737721 (+ 3363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.741077 (+ 3356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.742330 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.742566 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.742764 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.743050 (+ 286us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.743225 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13563,"thread_start_us":13230,"threads_started":1}
I20260812 11:27:40.743675 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.719589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:45440)
0812 11:27:40.731813 (+ 12224us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:40.731837 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:40.731994 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:40.735315 (+ 3321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:40.735320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:40.735358 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:40.736408 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.736428 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.737961 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.737974 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.740588 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.740605 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.742670 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.742679 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.742851 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.742883 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:27:40.742960 (+ 77us) client_negotiation.cc:240] Negotiation successful
0812 11:27:40.743226 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12043,"mutex_wait_us":113,"thread_start_us":14981,"threads_started":1}
I20260812 11:27:40.743947 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.723676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.12.110.194:45647)
0812 11:27:40.731208 (+ 7532us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.731223 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.734941 (+ 3718us) server_negotiation.cc:407] Connection header received
0812 11:27:40.735255 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.735262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.735479 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.735961 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:40.736846 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.738345 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.741094 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.742508 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.742717 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.742880 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.743422 (+ 542us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.743602 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7279,"thread_start_us":7095,"threads_started":1}
I20260812 11:27:40.744771 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.719038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38413 (local address 127.0.0.1:60554)
0812 11:27:40.728633 (+ 9595us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:40.728700 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:40.728984 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:40.733502 (+ 4518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:40.733510 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:40.733555 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:40.734091 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.734110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.737888 (+ 3778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.737896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.740853 (+ 2957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.740871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.742572 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.742581 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.742765 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.742823 (+ 58us) client_negotiation.cc:769] Sending connection context
0812 11:27:40.742993 (+ 170us) client_negotiation.cc:240] Negotiation successful
0812 11:27:40.744240 (+ 1247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9128,"mutex_wait_us":128,"thread_start_us":13049,"threads_started":1}
I20260812 11:27:40.744800 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.723413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:45647 (local address 127.0.0.1:53814)
0812 11:27:40.734802 (+ 11389us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:40.734822 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:40.735040 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:40.736004 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:40.736009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:40.736055 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:40.736669 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.736683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.738497 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.738502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.740965 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.740978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.743041 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.743047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.743221 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.743251 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:27:40.744168 (+ 917us) client_negotiation.cc:240] Negotiation successful
0812 11:27:40.744295 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11254,"mutex_wait_us":28,"thread_start_us":4489,"threads_started":1}
I20260812 11:27:40.745293 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.719975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45440 (local address 127.12.110.193:37869)
0812 11:27:40.734554 (+ 14579us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.734563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.734596 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:27:40.734797 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.734809 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.735069 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.736295 (+ 1226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:40.736688 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.737836 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.740778 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.742378 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.743216 (+ 838us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.743389 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.743451 (+ 62us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.744807 (+ 1356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14334,"thread_start_us":14121,"threads_started":1}
I20260812 11:27:40.748370 7571 tablet_service.cc:1467] Processing CreateTablet for tablet 48761b07f04645d89b565ea47ddce7ea (DEFAULT_TABLE table=test-table [id=fbd0cc0c748f46beacbeb23b87573a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:40.749759 7571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48761b07f04645d89b565ea47ddce7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:40.749539 7816 tablet_service.cc:1467] Processing CreateTablet for tablet 48761b07f04645d89b565ea47ddce7ea (DEFAULT_TABLE table=test-table [id=fbd0cc0c748f46beacbeb23b87573a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:40.749516 7344 tablet_service.cc:1467] Processing CreateTablet for tablet 48761b07f04645d89b565ea47ddce7ea (DEFAULT_TABLE table=test-table [id=fbd0cc0c748f46beacbeb23b87573a02]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:27:40.750967 7344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48761b07f04645d89b565ea47ddce7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:40.750965 7816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48761b07f04645d89b565ea47ddce7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:27:40.766803 7966 tablet_bootstrap.cc:492] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Bootstrap starting.
I20260812 11:27:40.769914 7968 tablet_bootstrap.cc:492] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Bootstrap starting.
I20260812 11:27:40.772141 7966 tablet_bootstrap.cc:654] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:40.773619 7968 tablet_bootstrap.cc:654] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:40.773640 7966 log.cc:824] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:40.774961 7969 tablet_bootstrap.cc:492] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Bootstrap starting.
I20260812 11:27:40.777544 7966 tablet_bootstrap.cc:492] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: No bootstrap required, opened a new log
I20260812 11:27:40.777853 7966 ts_tablet_manager.cc:1397] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 11:27:40.778487 7969 tablet_bootstrap.cc:654] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Neither blocks nor log segments found. Creating new log.
I20260812 11:27:40.779325 7968 log.cc:824] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:40.779965 7969 log.cc:824] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:27:40.784740 7968 tablet_bootstrap.cc:492] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: No bootstrap required, opened a new log
I20260812 11:27:40.784987 7969 tablet_bootstrap.cc:492] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: No bootstrap required, opened a new log
I20260812 11:27:40.785099 7968 ts_tablet_manager.cc:1397] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260812 11:27:40.785281 7969 ts_tablet_manager.cc:1397] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 11:27:40.790635 7966 raft_consensus.cc:348] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.791208 7966 raft_consensus.cc:374] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:40.791323 7966 raft_consensus.cc:729] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b3d5d1573446f683cf83e49f8d85d4, State: Initialized, Role: FOLLOWER
I20260812 11:27:40.792048 7966 consensus_queue.cc:260] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.792985 7968 raft_consensus.cc:348] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.793417 7968 raft_consensus.cc:374] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:40.793520 7968 raft_consensus.cc:729] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6523020d0bf4ddfbe9ad4c3befcb86d, State: Initialized, Role: FOLLOWER
I20260812 11:27:40.794495 7968 consensus_queue.cc:260] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.796523 7968 ts_tablet_manager.cc:1428] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260812 11:27:40.798269 7969 raft_consensus.cc:348] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.798933 7969 raft_consensus.cc:374] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:27:40.799073 7969 raft_consensus.cc:729] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f015c5d9844f7c82d84ccefa26e9a9, State: Initialized, Role: FOLLOWER
I20260812 11:27:40.799919 7969 consensus_queue.cc:260] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.801366 7927 heartbeater.cc:503] Master 127.12.110.254:42741 was elected leader, sending a full tablet report...
I20260812 11:27:40.801406 7696 heartbeater.cc:503] Master 127.12.110.254:42741 was elected leader, sending a full tablet report...
I20260812 11:27:40.802232 7966 ts_tablet_manager.cc:1428] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Time spent starting tablet: real 0.024s user 0.021s sys 0.004s
I20260812 11:27:40.803431 7969 ts_tablet_manager.cc:1428] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Time spent starting tablet: real 0.018s user 0.018s sys 0.000s
I20260812 11:27:40.825737 7975 raft_consensus.cc:482] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:27:40.826040 7975 raft_consensus.cc:504] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.827836 7975 leader_election.cc:288] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
I20260812 11:27:40.836336 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.828473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:50875 (local address 127.12.110.194:45647)
0812 11:27:40.828917 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.828928 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.829799 (+ 871us) server_negotiation.cc:407] Connection header received
0812 11:27:40.830181 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.830185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.830250 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.830305 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:40.831112 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.831981 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.834459 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.835283 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.835405 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.835467 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.835918 (+ 451us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.836063 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260812 11:27:40.836365 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.828397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:45647 (local address 127.12.110.193:50875)
0812 11:27:40.829658 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:40.829687 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:40.829962 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:40.830491 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:40.830494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:40.830576 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:40.830936 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.830945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.832154 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.832160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.834286 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.834299 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.835503 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.835513 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.835710 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.835769 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 11:27:40.835868 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:27:40.835996 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":749,"threads_started":1}
I20260812 11:27:40.836387 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.828421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:44129 (local address 127.12.110.195:38413)
0812 11:27:40.828672 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.828683 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.830336 (+ 1653us) server_negotiation.cc:407] Connection header received
0812 11:27:40.830570 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.830575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.830642 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.830743 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:27:40.831464 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.832343 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.834600 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.835418 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.835585 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.835668 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.836025 (+ 357us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.836162 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 11:27:40.836772 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.828437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38413 (local address 127.12.110.193:44129)
0812 11:27:40.830232 (+ 1795us) negotiation.cc:107] Waiting for socket to connect
0812 11:27:40.830246 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:27:40.830394 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:27:40.830765 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:27:40.830769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:27:40.830798 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:27:40.831252 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.831264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.832570 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.832577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.834386 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:27:40.834402 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.835647 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.835652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:27:40.835824 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.835875 (+ 51us) client_negotiation.cc:769] Sending connection context
0812 11:27:40.835981 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 11:27:40.836163 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1681,"thread_start_us":1460,"threads_started":1}
I20260812 11:27:40.837711 7849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48761b07f04645d89b565ea47ddce7ea" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" is_pre_election: true
I20260812 11:27:40.837605 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48761b07f04645d89b565ea47ddce7ea" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b3d5d1573446f683cf83e49f8d85d4" is_pre_election: true
I20260812 11:27:40.838240 7849 raft_consensus.cc:2454] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 0.
I20260812 11:27:40.838253 7618 raft_consensus.cc:2454] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 0.
I20260812 11:27:40.838954 7265 leader_election.cc:302] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f015c5d9844f7c82d84ccefa26e9a9, c6523020d0bf4ddfbe9ad4c3befcb86d; no voters:
I20260812 11:27:40.839381 7975 raft_consensus.cc:2781] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:27:40.839548 7975 raft_consensus.cc:482] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:27:40.839665 7975 raft_consensus.cc:3037] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:40.841948 7975 raft_consensus.cc:504] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.842763 7975 leader_election.cc:288] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 election: Requested vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
I20260812 11:27:40.843544 7849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48761b07f04645d89b565ea47ddce7ea" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9"
I20260812 11:27:40.843788 7618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48761b07f04645d89b565ea47ddce7ea" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b3d5d1573446f683cf83e49f8d85d4"
I20260812 11:27:40.843843 7849 raft_consensus.cc:3037] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:40.844038 7618 raft_consensus.cc:3037] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:27:40.846305 7618 raft_consensus.cc:2454] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 1.
I20260812 11:27:40.846483 7849 raft_consensus.cc:2454] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 1.
I20260812 11:27:40.846817 7268 leader_election.cc:302] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b3d5d1573446f683cf83e49f8d85d4, c6523020d0bf4ddfbe9ad4c3befcb86d; no voters:
I20260812 11:27:40.847170 7975 raft_consensus.cc:2781] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:27:40.847697 7975 raft_consensus.cc:686] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 LEADER]: Becoming Leader. State: Replica: c6523020d0bf4ddfbe9ad4c3befcb86d, State: Running, Role: LEADER
I20260812 11:27:40.848333 7975 consensus_queue.cc:237] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:27:40.853960 7114 catalog_manager.cc:5495] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6523020d0bf4ddfbe9ad4c3befcb86d (127.12.110.193). New cstate: current_term: 1 leader_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } health_report { overall_health: UNKNOWN } } }
W20260812 11:27:40.890661 7931 tablet.cc:2369] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:27:40.899502 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.892474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.12.110.193:37869)
0812 11:27:40.892689 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.892695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.893089 (+ 394us) server_negotiation.cc:407] Connection header received
0812 11:27:40.893262 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.893267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.893418 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.893633 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:27:40.894124 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.895011 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.896971 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.897430 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.897532 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.898080 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:27:40.898741 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:27:40.898859 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.899211 (+ 352us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.899291 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:27:40.908937 7618 raft_consensus.cc:1261] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Refusing update from remote peer c6523020d0bf4ddfbe9ad4c3befcb86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:40.908937 7849 raft_consensus.cc:1261] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Refusing update from remote peer c6523020d0bf4ddfbe9ad4c3befcb86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:27:40.910070 7983 consensus_queue.cc:1035] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:40.910492 7975 consensus_queue.cc:1035] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:27:40.922042 7987 mvcc.cc:205] Tried to move back new op lower bound from 7317643513471496192 to 7317643513238904832. Current Snapshot: MvccSnapshot[applied={T|T < 7317643513471496192}]
I20260812 11:27:40.938513 7112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:40.938933 7112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:27:40.942771 7112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Sending DeleteTablet for 3 replicas of tablet 48761b07f04645d89b565ea47ddce7ea
I20260812 11:27:40.943915 7816 tablet_service.cc:1514] Processing DeleteTablet for tablet 48761b07f04645d89b565ea47ddce7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:40 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:60554
I20260812 11:27:40.943972 7344 tablet_service.cc:1514] Processing DeleteTablet for tablet 48761b07f04645d89b565ea47ddce7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:40 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:45440
I20260812 11:27:40.944348 7571 tablet_service.cc:1514] Processing DeleteTablet for tablet 48761b07f04645d89b565ea47ddce7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:27:40 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:53814
I20260812 11:27:40.945617 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 7230
I20260812 11:27:40.959450 8012 tablet_replica.cc:330] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: stopping tablet replica
I20260812 11:27:40.959725 8009 tablet_replica.cc:330] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: stopping tablet replica
I20260812 11:27:40.960366 8014 tablet_replica.cc:330] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: stopping tablet replica
I20260812 11:27:40.960431 8012 raft_consensus.cc:2229] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:27:40.961061 8014 raft_consensus.cc:2229] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:40.961395 8009 raft_consensus.cc:2229] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:27:40.961508 8012 raft_consensus.cc:2258] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:40.962432 8014 raft_consensus.cc:2258] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:40.962779 8009 raft_consensus.cc:2258] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:27:40.964174 8012 ts_tablet_manager.cc:1905] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:40.965147 8009 ts_tablet_manager.cc:1905] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:27:40.966826 8014 ts_tablet_manager.cc:1905] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 11:27:40 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:27:40.973455 8012 ts_tablet_manager.cc:1918] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:40.973744 8012 log.cc:1196] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/48761b07f04645d89b565ea47ddce7ea
I20260812 11:27:40.974435 8012 ts_tablet_manager.cc:1939] T 48761b07f04645d89b565ea47ddce7ea P c6523020d0bf4ddfbe9ad4c3befcb86d: Deleting consensus metadata
I20260812 11:27:40.976507 7094 catalog_manager.cc:4841] TS c6523020d0bf4ddfbe9ad4c3befcb86d (127.12.110.193:37869): tablet 48761b07f04645d89b565ea47ddce7ea (table test-table [id=fbd0cc0c748f46beacbeb23b87573a02]) successfully deleted
I20260812 11:27:40.977063 8009 ts_tablet_manager.cc:1918] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:40.977263 8009 log.cc:1196] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/48761b07f04645d89b565ea47ddce7ea
I20260812 11:27:40.977782 8009 ts_tablet_manager.cc:1939] T 48761b07f04645d89b565ea47ddce7ea P a9f015c5d9844f7c82d84ccefa26e9a9: Deleting consensus metadata
I20260812 11:27:40.979426 7090 catalog_manager.cc:4841] TS a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413): tablet 48761b07f04645d89b565ea47ddce7ea (table test-table [id=fbd0cc0c748f46beacbeb23b87573a02]) successfully deleted
I20260812 11:27:40.979732 8014 ts_tablet_manager.cc:1918] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:27:40.979915 8014 log.cc:1196] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/48761b07f04645d89b565ea47ddce7ea
I20260812 11:27:40.980356 8014 ts_tablet_manager.cc:1939] T 48761b07f04645d89b565ea47ddce7ea P 42b3d5d1573446f683cf83e49f8d85d4: Deleting consensus metadata
I20260812 11:27:40.981690 7097 catalog_manager.cc:4841] TS 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647): tablet 48761b07f04645d89b565ea47ddce7ea (table test-table [id=fbd0cc0c748f46beacbeb23b87573a02]) successfully deleted
I20260812 11:27:40.983495 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:40.947078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.12.110.193:37869)
0812 11:27:40.947337 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:40.947347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:40.955340 (+ 7993us) server_negotiation.cc:407] Connection header received
0812 11:27:40.956267 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:40.956275 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:40.956375 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:40.956455 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:40.960383 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.961843 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.964528 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:40.965320 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:40.965431 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:40.976049 (+ 10618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:40.976065 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:40.976083 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:40.976145 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:40.980384 (+ 4239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.981320 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.981329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.981333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.981516 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:40.982003 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:40.982007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:40.982009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:40.982363 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:40.982683 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:40.983045 (+ 362us) server_negotiation.cc:299] Negotiation successful
0812 11:27:40.983224 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:27:40.984308 7448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:41.091285 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 7230
I20260812 11:27:41.119057 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 7466
Aug 12 11:27:41 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:27:41.137375 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:41.119946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.12.110.194:45647)
0812 11:27:41.120136 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:41.120140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:41.120159 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:27:41.120316 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:41.120320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:41.120395 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:41.120478 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:41.122080 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:41.122967 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:41.124777 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:41.125182 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:41.125278 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:41.130854 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:41.130871 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:41.130887 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:41.130927 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:41.133782 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:41.134778 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:41.134785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:41.134787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:41.134910 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:41.135502 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:41.135507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:41.135510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:41.136013 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:41.136302 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:41.136726 (+ 424us) server_negotiation.cc:299] Negotiation successful
0812 11:27:41.136965 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:27:41.138123 7688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:41.234858 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 7466
I20260812 11:27:41.262229 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 7704
Aug 12 11:27:41 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:27:41.281901 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:27:41.263193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.12.110.195:38413)
0812 11:27:41.263402 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:27:41.263411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:27:41.263438 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:27:41.263553 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:27:41.263558 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:27:41.263647 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:27:41.263761 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:27:41.265582 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:41.266500 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:41.267992 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:27:41.268688 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:27:41.268822 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:27:41.275346 (+ 6524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:27:41.275370 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:27:41.275395 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:27:41.275448 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:27:41.279212 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:41.279966 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:41.279973 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:41.279975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:41.280105 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:27:41.280447 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:27:41.280450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:27:41.280452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:27:41.280811 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:27:41.281038 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0812 11:27:41.281327 (+ 289us) server_negotiation.cc:299] Negotiation successful
0812 11:27:41.281516 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:27:41.282793 7918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:41.373735 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 7704
I20260812 11:27:41.392884 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 7063
I20260812 11:27:41.394655 7173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:27:41.510229 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 7063
I20260812 11:28:01.544549 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42741
--webserver_interface=127.12.110.254
--webserver_port=34839
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:01.912461 8029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:01.912840 8029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:01.913035 8029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:01.921195 8029 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:28:01.921360 8029 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:28:01.921469 8029 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:28:01.921581 8029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:28:01.921701 8029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:01.921840 8029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:01.921945 8029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:28:01.922040 8029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:28:01.932714 8029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=34839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:01.934106 8029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:01.935549 8029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:01.964488 8039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:01.964517 8043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:01.964773 8050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:01.965523 8029 server_base.cc:1034] running on GCE node
I20260812 11:28:01.966430 8029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:01.968171 8029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:01.969405 8029 hybrid_clock.cc:648] HybridClock initialized: now 1786534081969337 us; error 39 us; skew 500 ppm
I20260812 11:28:01.969761 8029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:01 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534081, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:01.977775 8029 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:28:01.979740 8029 webserver.cc:460] Webserver started at http://127.12.110.254:34839/ using document root <none> and password file <none>
I20260812 11:28:01.980500 8029 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:01.980615 8029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:01.986491 8029 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260812 11:28:01.989468 8058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:01.990845 8029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:01.990999 8029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "827b37303d194274b7bd989c63a22aab"
format_stamp: "Formatted at 2026-08-12 11:27:38 on dist-test-slave-h6n0"
server_key: "3db4854caaa4f53023fdc462a1d98ff2"
server_key_iv: "b57c5048ae13520a02f7fc57bb62cec2"
server_key_version: "encryptionkey@0"
I20260812 11:28:01.992036 8029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:02.026722 8029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:02.031121 8029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:02.031383 8029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:02.052338 8029 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:42741
I20260812 11:28:02.052374 8161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:42741 every 8 connection(s)
I20260812 11:28:02.053920 8029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:28:02.061025 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 8029
I20260812 11:28:02.061714 8162 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:28:02.061964 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:37869
--local_ip_for_outbound_sockets=127.12.110.193
--tserver_master_addrs=127.12.110.254:42741
--webserver_port=46107
--webserver_interface=127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:28:02.065343 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Bootstrap starting.
I20260812 11:28:02.091361 8162 log.cc:824] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:02.104460 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:28:02.105051 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Bootstrap complete.
I20260812 11:28:02.114702 8162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:02.115543 8162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827b37303d194274b7bd989c63a22aab, State: Initialized, Role: FOLLOWER
I20260812 11:28:02.116221 8162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:02.116479 8162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:28:02.116618 8162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:28:02.116808 8162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 1 FOLLOWER]: Advancing to term 2
I20260812 11:28:02.120599 8162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:02.121156 8162 leader_election.cc:302] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 827b37303d194274b7bd989c63a22aab; no voters:
I20260812 11:28:02.121770 8162 leader_election.cc:288] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [CANDIDATE]: Term 2 election: Requested vote from peers
I20260812 11:28:02.121981 8172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 FOLLOWER]: Leader election won for term 2
I20260812 11:28:02.123317 8172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 LEADER]: Becoming Leader. State: Replica: 827b37303d194274b7bd989c63a22aab, State: Running, Role: LEADER
I20260812 11:28:02.124017 8162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:28:02.123941 8172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:02.126938 8174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827b37303d194274b7bd989c63a22aab. Latest consensus state: current_term: 2 leader_uuid: "827b37303d194274b7bd989c63a22aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } } }
I20260812 11:28:02.127038 8173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "827b37303d194274b7bd989c63a22aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } } }
I20260812 11:28:02.127405 8174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:02.127456 8173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:02.128384 8181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:28:02.150883 8181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:28:02.153038 8181 catalog_manager.cc:1259] Loaded cluster ID: 58e328ac8b9843928778e12c03a50051
I20260812 11:28:02.153244 8181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:28:02.158959 8181 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:28:02.161108 8181 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Loaded TSK: 0
I20260812 11:28:02.204643 8181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Generated new TSK 1
I20260812 11:28:02.205068 8181 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260812 11:28:02.466995 8166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:02.467308 8166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:02.467479 8166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:02.475473 8166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:02.475630 8166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:02.475803 8166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:28:02.485663 8166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:37869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=46107
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:02.487046 8166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:02.488375 8166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:02.504453 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:02.504429 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:02.505072 8222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:02.505905 8166 server_base.cc:1034] running on GCE node
I20260812 11:28:02.506652 8166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:02.507637 8166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:02.508875 8166 hybrid_clock.cc:648] HybridClock initialized: now 1786534082508807 us; error 62 us; skew 500 ppm
I20260812 11:28:02.509199 8166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:02 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534082, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:02.512727 8166 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:28:02.514315 8166 webserver.cc:460] Webserver started at http://127.12.110.193:46107/ using document root <none> and password file <none>
I20260812 11:28:02.515079 8166 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:02.515249 8166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:02.520697 8166 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:28:02.523417 8246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:02.524915 8166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 11:28:02.525094 8166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "5ee8024bd1f348b27864dd16e35b9f26"
server_key_iv: "7238f6866241412ba3a98edb1305b66a"
server_key_version: "encryptionkey@0"
I20260812 11:28:02.526062 8166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:02.545634 8166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:02.549934 8166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:02.550237 8166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:02.551429 8166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:02.552923 8166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:02.553021 8166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:02.553188 8166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:02.553292 8166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:02.589298 8166 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:37869
I20260812 11:28:02.589352 8446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:37869 every 8 connection(s)
I20260812 11:28:02.590827 8166 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:28:02.592370 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 8166
I20260812 11:28:02.593273 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:45647
--local_ip_for_outbound_sockets=127.12.110.194
--tserver_master_addrs=127.12.110.254:42741
--webserver_port=40647
--webserver_interface=127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:28:02 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534082, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:02.621367 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:02.593955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:41885 (local address 127.12.110.254:42741)
0812 11:28:02.594693 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:02.594715 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:02.595391 (+ 676us) server_negotiation.cc:407] Connection header received
0812 11:28:02.598521 (+ 3130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:02.598603 (+ 82us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:02.598936 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:02.599225 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:02.602468 (+ 3243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:02.604097 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:02.606818 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:02.607552 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:02.607711 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:02.613176 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:02.613185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:02.613200 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:02.613243 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:02.616449 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:02.617522 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:02.617529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:02.617534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:02.617707 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:02.618287 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:02.618291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:02.618293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:02.618871 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:02.619033 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:02.620079 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0812 11:28:02.620406 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":282,"threads_started":1}
I20260812 11:28:02.622022 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:02.594064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:41885)
0812 11:28:02.595167 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:02.595230 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:02.597962 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:02.599930 (+ 1968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:02.599950 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:02.601256 (+ 1306us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:02.602072 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:02.602104 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:02.604387 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:02.604395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:02.606613 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:02.606633 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:02.607947 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:02.607958 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:02.608354 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:02.609400 (+ 1046us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:02.609443 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:02.612796 (+ 3353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:02.616616 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:02.616629 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:02.616646 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:02.617333 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:02.617860 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:02.617868 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:02.617874 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:02.618149 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:02.619084 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:02.619095 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:02.619539 (+ 444us) client_negotiation.cc:769] Sending connection context
0812 11:28:02.619868 (+ 329us) client_negotiation.cc:240] Negotiation successful
0812 11:28:02.620458 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":544,"threads_started":1}
I20260812 11:28:02.624889 8449 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42741
I20260812 11:28:02.625211 8449 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:02.625938 8449 heartbeater.cc:511] Master 127.12.110.254:42741 requested a full tablet report, sending...
I20260812 11:28:02.629473 8087 ts_manager.cc:194] Registered new tserver with Master: c6523020d0bf4ddfbe9ad4c3befcb86d (127.12.110.193:37869)
I20260812 11:28:02.631168 8087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:41885
W20260812 11:28:02.950995 8456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:02.951292 8456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:02.951450 8456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:02.958896 8456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:02.959000 8456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:02.959167 8456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:28:02.968994 8456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:45647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=40647
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:02.970317 8456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:02.971650 8456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:02.985119 8467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:02.985193 8472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:02.985067 8468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:02.986182 8456 server_base.cc:1034] running on GCE node
I20260812 11:28:02.986845 8456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:02.987757 8456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:02.988962 8456 hybrid_clock.cc:648] HybridClock initialized: now 1786534082988928 us; error 38 us; skew 500 ppm
I20260812 11:28:02.989270 8456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:02 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534082, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:02.992841 8456 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:28:02.994321 8456 webserver.cc:460] Webserver started at http://127.12.110.194:40647/ using document root <none> and password file <none>
I20260812 11:28:02.995048 8456 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:02.995221 8456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:03.000331 8456 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:28:03.003028 8489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:03.004303 8456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:03.004509 8456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "42b3d5d1573446f683cf83e49f8d85d4"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "490dc55451cb64a89310146c9def3872"
server_key_iv: "453b01f23afbf66db66cf4f7d38e65a3"
server_key_version: "encryptionkey@0"
I20260812 11:28:03.005481 8456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:03.019593 8456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:03.023422 8456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:03.023694 8456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:03.024677 8456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:03.025955 8456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:03.026084 8456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:03.026204 8456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:03.026293 8456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:03.064774 8456 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:45647
I20260812 11:28:03.064883 8694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:45647 every 8 connection(s)
I20260812 11:28:03.066701 8456 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:28:03.073560 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 8456
I20260812 11:28:03.074365 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:38413
--local_ip_for_outbound_sockets=127.12.110.195
--tserver_master_addrs=127.12.110.254:42741
--webserver_port=42089
--webserver_interface=127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 12 11:28:03 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534082, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:03.087745 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.069982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:36543 (local address 127.12.110.254:42741)
0812 11:28:03.070202 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.070208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.071450 (+ 1242us) server_negotiation.cc:407] Connection header received
0812 11:28:03.073580 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.073586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.073702 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.073826 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:03.075456 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.076194 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.077702 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.078143 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.078274 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.082082 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:03.082086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:03.082089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:03.082119 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:03.084752 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.085501 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.085507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.085512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.085635 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.086056 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.086061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.086065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.086346 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:03.086505 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.087186 (+ 681us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.087415 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:28:03.088747 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.070393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:36543)
0812 11:28:03.071282 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.071328 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:03.073296 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:03.074059 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:03.074066 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:03.074753 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:03.075278 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.075293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.076367 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.076371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.077542 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.077558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.078387 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.078394 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.078671 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.079375 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:03.079398 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:03.081860 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:03.084941 (+ 3081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:03.084950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:03.084964 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:03.085375 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:03.085745 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:03.085749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:03.085753 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:03.085936 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:03.086474 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:03.086481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:03.086821 (+ 340us) client_negotiation.cc:769] Sending connection context
0812 11:28:03.087082 (+ 261us) client_negotiation.cc:240] Negotiation successful
0812 11:28:03.087586 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":424,"threads_started":1}
I20260812 11:28:03.090207 8695 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42741
I20260812 11:28:03.090575 8695 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:03.091295 8695 heartbeater.cc:511] Master 127.12.110.254:42741 requested a full tablet report, sending...
I20260812 11:28:03.092844 8087 ts_manager.cc:194] Registered new tserver with Master: 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
I20260812 11:28:03.093984 8087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:36543
W20260812 11:28:03.416981 8698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:03.417281 8698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:03.417431 8698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:03.424762 8698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:03.424870 8698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:03.425064 8698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:28:03.435564 8698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:38413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=42089
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:03.436863 8698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:03.438091 8698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:03.451138 8717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:03.451686 8718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:03.452122 8723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:03.452491 8698 server_base.cc:1034] running on GCE node
I20260812 11:28:03.453121 8698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:03.454077 8698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:03.455322 8698 hybrid_clock.cc:648] HybridClock initialized: now 1786534083455261 us; error 42 us; skew 500 ppm
I20260812 11:28:03.455685 8698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:03 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534083, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:03.459215 8698 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:28:03.460824 8698 webserver.cc:460] Webserver started at http://127.12.110.195:42089/ using document root <none> and password file <none>
I20260812 11:28:03.461556 8698 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:03.461736 8698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:03.467314 8698 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:28:03.470149 8739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:03.471515 8698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:28:03.471722 8698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "a9f015c5d9844f7c82d84ccefa26e9a9"
format_stamp: "Formatted at 2026-08-12 11:27:40 on dist-test-slave-h6n0"
server_key: "3a5cbf03cf6126ae59b0a5f434aaeba9"
server_key_iv: "782c4b9d7f4c7aa28eb92f49ba6ac1f7"
server_key_version: "encryptionkey@0"
I20260812 11:28:03.472697 8698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:03.490833 8698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:03.497567 8698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:03.497931 8698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:03.499082 8698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:03.500566 8698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:03.500706 8698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:03.500850 8698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:03.500941 8698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:03.537501 8698 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:38413
I20260812 11:28:03.537606 8952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:38413 every 8 connection(s)
I20260812 11:28:03.539705 8698 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:28:03.549280 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 8698
Aug 12 11:28:03 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534083, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:03.563942 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.542449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:40113 (local address 127.12.110.254:42741)
0812 11:28:03.542712 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.542718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.544231 (+ 1513us) server_negotiation.cc:407] Connection header received
0812 11:28:03.546887 (+ 2656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.546892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.546984 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.547093 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:03.548714 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.549554 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.551421 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.552195 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.552336 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.556427 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:03.556436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:03.556446 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:03.556478 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:03.560509 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.561216 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.561223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.561232 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.561403 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.561914 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.561919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.561925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.562317 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:03.562459 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.563287 (+ 828us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.563531 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260812 11:28:03.564733 8961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.542961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:40113)
0812 11:28:03.543957 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.544048 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:03.546422 (+ 2374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:03.547380 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:03.547388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:03.547933 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:03.548531 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.548558 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.549736 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.549741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.551267 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.551283 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.552430 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.552435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.552643 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.553360 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:03.553382 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:03.556208 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:03.560660 (+ 4452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:03.560666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:03.560678 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:03.561101 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:03.561544 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:03.561550 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:03.561555 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:03.561773 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:03.562468 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:03.562477 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:03.562888 (+ 411us) client_negotiation.cc:769] Sending connection context
0812 11:28:03.563123 (+ 235us) client_negotiation.cc:240] Negotiation successful
0812 11:28:03.563492 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":286,"threads_started":1}
I20260812 11:28:03.567166 8957 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42741
I20260812 11:28:03.567724 8957 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:03.568470 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.550253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.12.110.254:42741)
0812 11:28:03.550772 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.550778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.551169 (+ 391us) server_negotiation.cc:407] Connection header received
0812 11:28:03.551312 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.551319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.551417 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.551518 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:03.553611 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.554723 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.556437 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.556985 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.557134 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.560625 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:03.560644 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:03.560648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:03.560686 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:03.564055 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.565126 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.565131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.565134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.565267 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.566107 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.566110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.566112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.566609 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:03.566817 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.567460 (+ 643us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.567848 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":290,"threads_started":1}
I20260812 11:28:03.569108 8957 heartbeater.cc:511] Master 127.12.110.254:42741 requested a full tablet report, sending...
I20260812 11:28:03.571291 8085 ts_manager.cc:194] Registered new tserver with Master: a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413)
I20260812 11:28:03.572425 8085 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:40113
I20260812 11:28:03.584146 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:28:03.591148 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.585131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.12.110.254:42741)
0812 11:28:03.585327 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.585331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.585781 (+ 450us) server_negotiation.cc:407] Connection header received
0812 11:28:03.585958 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.585962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.586126 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.586261 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:28:03.586750 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.587524 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.589111 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.589537 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.589678 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.590169 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:28:03.590428 (+ 259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0812 11:28:03.590845 (+ 417us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56830: token expired
Metrics: {"server-negotiator.queue_time_us":76}
W20260812 11:28:03.591290 8962 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56830: token expired
W20260812 11:28:03.591920 7938 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.12.110.254:42741)
I20260812 11:28:03.606799 8962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.592621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.12.110.254:42741)
0812 11:28:03.592827 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.592843 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.592864 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:28:03.592910 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.592915 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.592986 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.593078 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:03.594475 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.595457 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.597157 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.597531 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.597632 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.600835 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:03.600839 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:03.600842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:03.600872 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:03.603946 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.604690 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.604698 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.604701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.604872 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.605262 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.605267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.605271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.605598 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:03.605864 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.606210 (+ 346us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.606438 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:28:03.610306 8085 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:28:03.618322 8085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:28:03.634588 8449 heartbeater.cc:503] Master 127.12.110.254:42741 was elected leader, sending a full tablet report...
I20260812 11:28:03.653362 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.630648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.12.110.193:37869)
0812 11:28:03.644855 (+ 14207us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.644866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.644901 (+ 35us) server_negotiation.cc:407] Connection header received
0812 11:28:03.645127 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.645137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.645339 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.645591 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:03.647027 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.648557 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.650999 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.652145 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.652388 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.652574 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.652857 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.653033 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14020,"thread_start_us":13889,"threads_started":1}
I20260812 11:28:03.653469 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.630266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60122)
0812 11:28:03.637269 (+ 7003us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.637319 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:03.637538 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:03.645633 (+ 8095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:03.645643 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:03.645687 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:03.646774 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.646797 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.648726 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.648730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.650803 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.650820 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.652395 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.652402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.652611 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.652665 (+ 54us) client_negotiation.cc:769] Sending connection context
0812 11:28:03.652795 (+ 130us) client_negotiation.cc:240] Negotiation successful
0812 11:28:03.653006 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6681,"mutex_wait_us":125,"thread_start_us":6510,"threads_started":1}
I20260812 11:28:03.654335 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.630932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:45647 (local address 127.0.0.1:47678)
0812 11:28:03.642207 (+ 11275us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.642252 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:03.642591 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:03.645923 (+ 3332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:03.645931 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:03.645964 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:03.646860 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.646878 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.649242 (+ 2364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.649247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.651378 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.651394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.653490 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.653496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.653668 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.653701 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:28:03.653786 (+ 85us) client_negotiation.cc:240] Negotiation successful
0812 11:28:03.653927 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11023,"mutex_wait_us":7,"thread_start_us":15415,"threads_started":1}
I20260812 11:28:03.654677 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.631525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.12.110.194:45647)
0812 11:28:03.644069 (+ 12544us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.644079 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.644112 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:28:03.644219 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.644228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.644500 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.644723 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:03.647065 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.648881 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.651561 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.652985 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.653160 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.653310 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.653970 (+ 660us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.654151 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12337,"thread_start_us":12630,"threads_started":1}
I20260812 11:28:03.655723 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.631084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38413 (local address 127.0.0.1:52216)
0812 11:28:03.646812 (+ 15728us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.646839 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:03.646984 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:03.647884 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:03.647889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:03.647914 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:03.648303 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.648316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.650241 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.650246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.652189 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.652205 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.654934 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.654942 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.655149 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.655182 (+ 33us) client_negotiation.cc:769] Sending connection context
0812 11:28:03.655269 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 11:28:03.655400 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15505,"thread_start_us":10569,"threads_started":1}
I20260812 11:28:03.656659 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.630865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52216 (local address 127.12.110.195:38413)
0812 11:28:03.645601 (+ 14736us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.645608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.647102 (+ 1494us) server_negotiation.cc:407] Connection header received
0812 11:28:03.647210 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.647217 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.647419 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.647605 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:03.648499 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.650090 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.652387 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.653830 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.655856 (+ 2026us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.656024 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.656102 (+ 78us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.656263 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14520,"thread_start_us":14323,"threads_started":1}
I20260812 11:28:03.657976 8328 tablet_service.cc:1467] Processing CreateTablet for tablet d0e273ceeefc49cfa7d97df14dc2ccca (DEFAULT_TABLE table=test-table [id=ea0c16c101184266bb6636c683dffbc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:03.658125 8578 tablet_service.cc:1467] Processing CreateTablet for tablet d0e273ceeefc49cfa7d97df14dc2ccca (DEFAULT_TABLE table=test-table [id=ea0c16c101184266bb6636c683dffbc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:03.659397 8578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e273ceeefc49cfa7d97df14dc2ccca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:03.659581 8328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e273ceeefc49cfa7d97df14dc2ccca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:03.660370 8828 tablet_service.cc:1467] Processing CreateTablet for tablet d0e273ceeefc49cfa7d97df14dc2ccca (DEFAULT_TABLE table=test-table [id=ea0c16c101184266bb6636c683dffbc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:03.661695 8828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0e273ceeefc49cfa7d97df14dc2ccca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:03.672595 8988 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Bootstrap starting.
I20260812 11:28:03.676003 8988 tablet_bootstrap.cc:654] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:03.676728 8989 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Bootstrap starting.
I20260812 11:28:03.677578 8988 log.cc:824] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:03.679096 8994 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Bootstrap starting.
I20260812 11:28:03.680579 8988 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: No bootstrap required, opened a new log
I20260812 11:28:03.680720 8989 tablet_bootstrap.cc:654] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:03.680894 8988 ts_tablet_manager.cc:1397] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260812 11:28:03.681944 8989 log.cc:824] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:03.682626 8994 tablet_bootstrap.cc:654] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:03.683787 8994 log.cc:824] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:03.685559 8989 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: No bootstrap required, opened a new log
I20260812 11:28:03.685845 8989 ts_tablet_manager.cc:1397] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260812 11:28:03.687026 8994 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: No bootstrap required, opened a new log
I20260812 11:28:03.687290 8994 ts_tablet_manager.cc:1397] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260812 11:28:03.689428 8988 raft_consensus.cc:348] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.689800 8988 raft_consensus.cc:374] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:03.689924 8988 raft_consensus.cc:729] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b3d5d1573446f683cf83e49f8d85d4, State: Initialized, Role: FOLLOWER
I20260812 11:28:03.690575 8988 consensus_queue.cc:260] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.691671 8695 heartbeater.cc:503] Master 127.12.110.254:42741 was elected leader, sending a full tablet report...
I20260812 11:28:03.692302 8988 ts_tablet_manager.cc:1428] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Time spent starting tablet: real 0.011s user 0.003s sys 0.009s
I20260812 11:28:03.696691 8989 raft_consensus.cc:348] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.697144 8989 raft_consensus.cc:374] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:03.697247 8989 raft_consensus.cc:729] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6523020d0bf4ddfbe9ad4c3befcb86d, State: Initialized, Role: FOLLOWER
I20260812 11:28:03.697706 8994 raft_consensus.cc:348] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.697854 8989 consensus_queue.cc:260] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.698105 8994 raft_consensus.cc:374] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:03.698210 8994 raft_consensus.cc:729] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f015c5d9844f7c82d84ccefa26e9a9, State: Initialized, Role: FOLLOWER
I20260812 11:28:03.698900 8994 consensus_queue.cc:260] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.700210 8957 heartbeater.cc:503] Master 127.12.110.254:42741 was elected leader, sending a full tablet report...
I20260812 11:28:03.701072 8994 ts_tablet_manager.cc:1428] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260812 11:28:03.702625 8989 ts_tablet_manager.cc:1428] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Time spent starting tablet: real 0.017s user 0.011s sys 0.009s
I20260812 11:28:03.709110 8999 raft_consensus.cc:482] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:03.709354 8999 raft_consensus.cc:504] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.711059 8999 leader_election.cc:288] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
I20260812 11:28:03.721632 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.711873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:56887 (local address 127.12.110.195:38413)
0812 11:28:03.712078 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.712087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.712758 (+ 671us) server_negotiation.cc:407] Connection header received
0812 11:28:03.712928 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.712933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.713045 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.713201 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:03.713966 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.715066 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.719085 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.720359 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.720510 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.720574 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.721226 (+ 652us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.721377 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260812 11:28:03.721637 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.711517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:43105 (local address 127.12.110.194:45647)
0812 11:28:03.711768 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.711775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.713315 (+ 1540us) server_negotiation.cc:407] Connection header received
0812 11:28:03.713600 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.713604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.713669 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.713750 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:03.714286 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.715331 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.719091 (+ 3760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.720324 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.720500 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.720574 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.721226 (+ 652us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.721369 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:28:03.721879 9006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.711486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38413 (local address 127.12.110.193:56887)
0812 11:28:03.712552 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.712568 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:03.712684 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:03.713294 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:03.713300 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:03.713328 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:03.713771 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.713782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.715301 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.715306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.718886 (+ 3580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.718904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.720661 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.720670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.720942 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.721070 (+ 128us) client_negotiation.cc:769] Sending connection context
0812 11:28:03.721169 (+ 99us) client_negotiation.cc:240] Negotiation successful
0812 11:28:03.721368 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":241,"thread_start_us":1428,"threads_started":1}
I20260812 11:28:03.721899 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.711511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:45647 (local address 127.12.110.193:43105)
0812 11:28:03.713209 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.713238 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:03.713411 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:03.713793 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:03.713797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:03.713824 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:03.714120 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.714128 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.715486 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.715494 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.718886 (+ 3392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:03.718904 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.720681 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.720686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:03.720939 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.721063 (+ 124us) client_negotiation.cc:769] Sending connection context
0812 11:28:03.721169 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 11:28:03.721369 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"mutex_wait_us":196,"spinlock_wait_cycles":18688,"thread_start_us":691,"threads_started":1}
I20260812 11:28:03.722931 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e273ceeefc49cfa7d97df14dc2ccca" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" is_pre_election: true
I20260812 11:28:03.722929 8617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e273ceeefc49cfa7d97df14dc2ccca" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b3d5d1573446f683cf83e49f8d85d4" is_pre_election: true
I20260812 11:28:03.723407 8617 raft_consensus.cc:2454] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 0.
I20260812 11:28:03.723402 8854 raft_consensus.cc:2454] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 0.
I20260812 11:28:03.724135 8250 leader_election.cc:302] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f015c5d9844f7c82d84ccefa26e9a9, c6523020d0bf4ddfbe9ad4c3befcb86d; no voters:
I20260812 11:28:03.724551 8999 raft_consensus.cc:2781] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:28:03.724704 8999 raft_consensus.cc:482] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:28:03.724817 8999 raft_consensus.cc:3037] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:03.727090 8999 raft_consensus.cc:504] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.727764 8999 leader_election.cc:288] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 election: Requested vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
I20260812 11:28:03.728103 8854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e273ceeefc49cfa7d97df14dc2ccca" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9"
I20260812 11:28:03.728298 8854 raft_consensus.cc:3037] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:03.728291 8617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0e273ceeefc49cfa7d97df14dc2ccca" candidate_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b3d5d1573446f683cf83e49f8d85d4"
I20260812 11:28:03.728492 8617 raft_consensus.cc:3037] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:03.730880 8854 raft_consensus.cc:2454] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 1.
I20260812 11:28:03.731200 8617 raft_consensus.cc:2454] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6523020d0bf4ddfbe9ad4c3befcb86d in term 1.
I20260812 11:28:03.731348 8250 leader_election.cc:302] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9f015c5d9844f7c82d84ccefa26e9a9, c6523020d0bf4ddfbe9ad4c3befcb86d; no voters:
I20260812 11:28:03.731662 8999 raft_consensus.cc:2781] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:03.733342 8999 raft_consensus.cc:686] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 LEADER]: Becoming Leader. State: Replica: c6523020d0bf4ddfbe9ad4c3befcb86d, State: Running, Role: LEADER
I20260812 11:28:03.733861 8999 consensus_queue.cc:237] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:03.738751 8085 catalog_manager.cc:5495] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d reported cstate change: term changed from 0 to 1, leader changed from <none> to c6523020d0bf4ddfbe9ad4c3befcb86d (127.12.110.193). New cstate: current_term: 1 leader_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } health_report { overall_health: UNKNOWN } } }
I20260812 11:28:03.769336 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.763769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.12.110.193:37869)
0812 11:28:03.764035 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.764040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.764056 (+ 16us) server_negotiation.cc:407] Connection header received
0812 11:28:03.764145 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.764149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.764236 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.764402 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:28:03.764779 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.765523 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.767134 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.767547 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.767683 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.768202 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:28:03.768708 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:28:03.768867 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.768981 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.769071 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:28:03.777767 8854 raft_consensus.cc:1261] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Refusing update from remote peer c6523020d0bf4ddfbe9ad4c3befcb86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:03.778348 8617 raft_consensus.cc:1261] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Refusing update from remote peer c6523020d0bf4ddfbe9ad4c3befcb86d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:03.778926 9008 consensus_queue.cc:1035] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:03.779760 8999 consensus_queue.cc:1035] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:28:03.791388 8960 tablet.cc:2369] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:28:03.792481 9013 mvcc.cc:205] Tried to move back new op lower bound from 7317643607143395328 to 7317643606977671168. Current Snapshot: MvccSnapshot[applied={T|T < 7317643607143395328}]
I20260812 11:28:03.793275 9016 mvcc.cc:205] Tried to move back new op lower bound from 7317643607143395328 to 7317643606977671168. Current Snapshot: MvccSnapshot[applied={T|T < 7317643607143395328}]
I20260812 11:28:03.807905 8085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:03.808233 8085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:03.812059 8085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Sending DeleteTablet for 3 replicas of tablet d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:03.813305 8328 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e273ceeefc49cfa7d97df14dc2ccca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:03 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:60122
I20260812 11:28:03.813366 8828 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e273ceeefc49cfa7d97df14dc2ccca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:03 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:52216
I20260812 11:28:03.813591 8578 tablet_service.cc:1514] Processing DeleteTablet for tablet d0e273ceeefc49cfa7d97df14dc2ccca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:03 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:47678
I20260812 11:28:03.814246 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 8166
I20260812 11:28:03.817886 9032 tablet_replica.cc:330] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: stopping tablet replica
I20260812 11:28:03.818416 9032 raft_consensus.cc:2229] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:03.819105 9032 raft_consensus.cc:2258] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:03.823647 9032 ts_tablet_manager.cc:1905] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:03.828662 9030 tablet_replica.cc:330] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: stopping tablet replica
I20260812 11:28:03.829257 9030 raft_consensus.cc:2229] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:03.829953 9030 raft_consensus.cc:2258] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:03.831171 9032 ts_tablet_manager.cc:1918] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:03.831398 9032 log.cc:1196] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:03.832078 9032 ts_tablet_manager.cc:1939] T d0e273ceeefc49cfa7d97df14dc2ccca P 42b3d5d1573446f683cf83e49f8d85d4: Deleting consensus metadata
I20260812 11:28:03.832407 9030 ts_tablet_manager.cc:1905] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:03.834052 8063 catalog_manager.cc:4841] TS 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647): tablet d0e273ceeefc49cfa7d97df14dc2ccca (table test-table [id=ea0c16c101184266bb6636c683dffbc0]) successfully deleted
I20260812 11:28:03.837682 9030 ts_tablet_manager.cc:1918] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:03.837857 9030 log.cc:1196] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:03.838495 9030 ts_tablet_manager.cc:1939] T d0e273ceeefc49cfa7d97df14dc2ccca P a9f015c5d9844f7c82d84ccefa26e9a9: Deleting consensus metadata
I20260812 11:28:03.839918 9031 tablet_replica.cc:330] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: stopping tablet replica
I20260812 11:28:03.840389 9031 raft_consensus.cc:2229] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:28:03.840412 8060 catalog_manager.cc:4841] TS a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413): tablet d0e273ceeefc49cfa7d97df14dc2ccca (table test-table [id=ea0c16c101184266bb6636c683dffbc0]) successfully deleted
I20260812 11:28:03.841128 9031 raft_consensus.cc:2258] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:03.843021 9031 ts_tablet_manager.cc:1905] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:03.843751 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.815173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.12.110.193:37869)
0812 11:28:03.815397 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.815406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.815432 (+ 26us) server_negotiation.cc:407] Connection header received
0812 11:28:03.815564 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.815572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.815677 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.815826 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:03.817724 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.819404 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.821257 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.822178 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.822344 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.826685 (+ 4341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:03.826704 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:03.826726 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:03.826796 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:03.839300 (+ 12504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.840231 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.840240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.840244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.840443 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.840948 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.840956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.840959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.841457 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:03.842880 (+ 1423us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.843061 (+ 181us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.843333 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:28:03.844575 8439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:03.948133 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 8166
W20260812 11:28:03.974429 8062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.110.193:37869 (error 108)
I20260812 11:28:03.974984 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 8456
I20260812 11:28:03.976788 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.976291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60142)
0812 11:28:03.976464 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:03.976582 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260812 11:28:03.977035 8062 catalog_manager.cc:4584] TS c6523020d0bf4ddfbe9ad4c3befcb86d (127.12.110.193:37869): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0e273ceeefc49cfa7d97df14dc2ccca: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
I20260812 11:28:03.989446 8986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:03.975724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.12.110.194:45647)
0812 11:28:03.975929 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:03.975933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:03.976054 (+ 121us) server_negotiation.cc:407] Connection header received
0812 11:28:03.976233 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:03.976239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:03.976383 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:03.976509 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:03.978021 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.978960 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.980294 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:03.980706 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:03.980848 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:03.983705 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:03.983726 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:03.983741 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:03.983777 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:03.986508 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.987325 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.987335 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.987338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.987480 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:03.987872 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:03.987877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:03.987880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:03.988265 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:03.988601 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:03.988882 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 11:28:03.989106 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260812 11:28:03.990079 8688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:04.030925 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.030301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60146)
0812 11:28:04.030582 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:04.030709 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260812 11:28:04.031991 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.031600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60150)
0812 11:28:04.031755 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:04.031841 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 11:28:04.092197 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 8456
I20260812 11:28:04.092619 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.092084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60156)
0812 11:28:04.092339 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:04.092444 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 11:28:04.094295 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.093961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60172)
0812 11:28:04.094106 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:04.094186 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:28:04.113083 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 8698
I20260812 11:28:04.128033 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.114040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.12.110.195:38413)
0812 11:28:04.114239 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:04.114245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:04.114266 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:28:04.114449 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:04.114454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:04.114668 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:04.114780 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:04.116206 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:04.116954 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:04.118245 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:04.118718 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:04.118854 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:04.121680 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:04.121694 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:04.121711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:04.121743 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:04.124728 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:04.125485 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:04.125496 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:04.125499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:04.125649 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:04.126165 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:04.126172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:04.126176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:04.126673 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:04.126999 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:04.127388 (+ 389us) server_negotiation.cc:299] Negotiation successful
0812 11:28:04.127629 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:28:04.128851 8939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:04.175182 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.174519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60174)
0812 11:28:04.174796 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:04.174954 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 11:28:04.176576 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.176078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60188)
0812 11:28:04.176300 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:04.176400 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 11:28:04.240058 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 8698
I20260812 11:28:04.265269 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 8029
I20260812 11:28:04.266456 8152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:04.389989 8983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:04.389151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:37869 (local address 127.0.0.1:60194)
0812 11:28:04.389666 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:04.389800 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:37869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 11:28:04.390010 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 8029
I20260812 11:28:24.422950 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42741
--webserver_interface=127.12.110.254
--webserver_port=34839
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:42741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:24.788866 9050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:24.789346 9050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:24.789549 9050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:24.801648 9050 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260812 11:28:24.801817 9050 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260812 11:28:24.801882 9050 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260812 11:28:24.801970 9050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:28:24.802042 9050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:24.802131 9050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:24.802196 9050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:28:24.802271 9050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:28:24.815294 9050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:42741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=34839
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:24.816705 9050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:24.818125 9050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:24.830858 9062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:24.830981 9060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:24.830951 9059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:24.832388 9050 server_base.cc:1034] running on GCE node
I20260812 11:28:24.833271 9050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:24.834741 9050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:24.835991 9050 hybrid_clock.cc:648] HybridClock initialized: now 1786534104835923 us; error 38 us; skew 500 ppm
I20260812 11:28:24.836333 9050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:24 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534104, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:24.839490 9050 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:28:24.840993 9050 webserver.cc:460] Webserver started at http://127.12.110.254:34839/ using document root <none> and password file <none>
I20260812 11:28:24.841686 9050 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:24.841825 9050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:24.847683 9050 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260812 11:28:24.850826 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:24.852283 9050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260812 11:28:24.852500 9050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "827b37303d194274b7bd989c63a22aab"
format_stamp: "Formatted at 2026-08-12 11:27:38 on dist-test-slave-h6n0"
server_key: "3db4854caaa4f53023fdc462a1d98ff2"
server_key_iv: "b57c5048ae13520a02f7fc57bb62cec2"
server_key_version: "encryptionkey@0"
I20260812 11:28:24.853526 9050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:24.872536 9050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:24.876839 9050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:24.877162 9050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:24.898483 9050 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:42741
I20260812 11:28:24.898576 9173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:42741 every 8 connection(s)
I20260812 11:28:24.900144 9050 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:28:24.907419 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 9050
I20260812 11:28:24.908566 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:37869
--local_ip_for_outbound_sockets=127.12.110.193
--tserver_master_addrs=127.12.110.254:42741
--webserver_port=46107
--webserver_interface=127.12.110.193
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:28:24.909265 9174 sys_catalog.cc:263] Verifying existing consensus state
I20260812 11:28:24.914489 9174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Bootstrap starting.
I20260812 11:28:24.938316 9174 log.cc:824] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:24.955106 9174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:28:24.955659 9174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Bootstrap complete.
I20260812 11:28:24.965687 9174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:24.966638 9174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 827b37303d194274b7bd989c63a22aab, State: Initialized, Role: FOLLOWER
I20260812 11:28:24.967249 9174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:24.967510 9174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:28:24.967643 9174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:28:24.967840 9174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 2 FOLLOWER]: Advancing to term 3
I20260812 11:28:24.971616 9174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:24.972515 9174 leader_election.cc:302] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 827b37303d194274b7bd989c63a22aab; no voters:
I20260812 11:28:24.973348 9174 leader_election.cc:288] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [CANDIDATE]: Term 3 election: Requested vote from peers
I20260812 11:28:24.973480 9178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 3 FOLLOWER]: Leader election won for term 3
I20260812 11:28:24.975811 9174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:28:24.978942 9178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [term 3 LEADER]: Becoming Leader. State: Replica: 827b37303d194274b7bd989c63a22aab, State: Running, Role: LEADER
I20260812 11:28:24.979624 9178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } }
I20260812 11:28:24.982553 9180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "827b37303d194274b7bd989c63a22aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } } }
I20260812 11:28:24.982985 9180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:24.983601 9181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 827b37303d194274b7bd989c63a22aab. Latest consensus state: current_term: 3 leader_uuid: "827b37303d194274b7bd989c63a22aab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "827b37303d194274b7bd989c63a22aab" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42741 } } }
I20260812 11:28:24.983938 9181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:24.995348 9198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:28:25.004912 9198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:28:25.006568 9198 catalog_manager.cc:1259] Loaded cluster ID: 58e328ac8b9843928778e12c03a50051
I20260812 11:28:25.006752 9198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:28:25.011615 9216 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 11:28:25.012035 9198 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:28:25.013842 9198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Loaded TSK: 0
I20260812 11:28:25.013983 9198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 827b37303d194274b7bd989c63a22aab: Loaded TSK: 1
F20260812 11:28:25.015070 9198 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f45189019ad google::LogMessage::Fail() at ??:0
@ 0x7f451890488a google::LogMessage::SendToLog() at ??:0
@ 0x7f45189014de google::LogMessage::Flush() at ??:0
@ 0x7f4518902a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f451c6657d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f451c65473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f452e3301c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f452e32f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f452e3bb2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f452e3bb0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f452e4055e9 std::function<>::operator()() at ??:0
@ 0x7f452e333cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f452e331c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f452e3b9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f452e3b9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f451b54d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f451b55091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f451b550799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f451b52beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f451e8056db start_thread at ??:0
@ 0x7f451779b61f clone at ??:0
W20260812 11:28:25.331787 9176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:25.332067 9176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:25.332243 9176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:25.339609 9176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:25.339710 9176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:25.339933 9176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:28:25.349536 9176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:37869
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=46107
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:25.350806 9176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:25.352128 9176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:25.367228 9260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:25.367259 9261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:25.367323 9263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:25.368297 9176 server_base.cc:1034] running on GCE node
I20260812 11:28:25.369014 9176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:25.369982 9176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:25.371214 9176 hybrid_clock.cc:648] HybridClock initialized: now 1786534105371177 us; error 58 us; skew 500 ppm
I20260812 11:28:25.371531 9176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:25 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534105, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:25.374963 9176 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:28:25.376361 9176 webserver.cc:460] Webserver started at http://127.12.110.193:46107/ using document root <none> and password file <none>
I20260812 11:28:25.377193 9176 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:25.377318 9176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:25.382766 9176 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:25.385501 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:25.386793 9176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:25.386940 9176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "5ee8024bd1f348b27864dd16e35b9f26"
server_key_iv: "7238f6866241412ba3a98edb1305b66a"
server_key_version: "encryptionkey@0"
I20260812 11:28:25.387844 9176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:25.410358 9176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:25.414625 9176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:25.414871 9176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:25.415782 9176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:25.417606 9290 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260812 11:28:25.423300 9176 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260812 11:28:25.423712 9176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260812 11:28:25.423897 9176 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260812 11:28:25.427016 9176 ts_tablet_manager.cc:610] Registered 1 tablets
I20260812 11:28:25.427103 9176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:25.427544 9290 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Bootstrap starting.
I20260812 11:28:25.436800 9290 log.cc:824] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:25.451025 9290 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260812 11:28:25.451951 9290 tablet_bootstrap.cc:492] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Bootstrap complete.
I20260812 11:28:25.452632 9290 ts_tablet_manager.cc:1397] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.008s
I20260812 11:28:25.463405 9290 raft_consensus.cc:348] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:25.465009 9290 raft_consensus.cc:729] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6523020d0bf4ddfbe9ad4c3befcb86d, State: Initialized, Role: FOLLOWER
I20260812 11:28:25.465755 9290 consensus_queue.cc:260] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:25.467715 9290 ts_tablet_manager.cc:1428] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260812 11:28:25.480847 9176 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:37869
I20260812 11:28:25.480880 9485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:37869 every 8 connection(s)
I20260812 11:28:25.482218 9176 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:28:25.486327 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 9176
I20260812 11:28:25.487119 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:45647
--local_ip_for_outbound_sockets=127.12.110.194
--tserver_master_addrs=127.12.110.254:42741
--webserver_port=40647
--webserver_interface=127.12.110.194
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:28:25.487350 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:25.485201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:37017)
0812 11:28:25.486074 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:25.486315 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":405,"threads_started":1}
W20260812 11:28:25.487977 9486 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:42741 (0 consecutive failures): Network error: Failed to ping master at 127.12.110.254:42741: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
I20260812 11:28:25.489002 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:25.488646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:55261)
0812 11:28:25.488811 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:25.488899 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260812 11:28:25.489908 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:25.489558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:59487)
0812 11:28:25.489710 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:25.489791 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 11:28:25.490136 9486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260812 11:28:25.852964 9490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:25.853269 9490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:25.853461 9490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:25.861375 9490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:25.861485 9490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:25.861680 9490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:28:25.872486 9490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:45647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=40647
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:25.873862 9490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:25.875255 9490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:25.888440 9497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:25.888486 9501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:25.888574 9499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:25.889760 9490 server_base.cc:1034] running on GCE node
I20260812 11:28:25.890367 9490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:25.891350 9490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:25.892583 9490 hybrid_clock.cc:648] HybridClock initialized: now 1786534105892547 us; error 48 us; skew 500 ppm
I20260812 11:28:25.892903 9490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:25 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534105, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:25.896360 9490 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:28:25.897799 9490 webserver.cc:460] Webserver started at http://127.12.110.194:40647/ using document root <none> and password file <none>
I20260812 11:28:25.898487 9490 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:25.898671 9490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:25.904338 9490 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:28:25.907706 9516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:25.909029 9490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:28:25.909237 9490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "42b3d5d1573446f683cf83e49f8d85d4"
format_stamp: "Formatted at 2026-08-12 11:27:39 on dist-test-slave-h6n0"
server_key: "490dc55451cb64a89310146c9def3872"
server_key_iv: "453b01f23afbf66db66cf4f7d38e65a3"
server_key_version: "encryptionkey@0"
I20260812 11:28:25.910333 9490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:25.927170 9490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:25.931516 9490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:25.931823 9490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:25.932832 9490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:25.934226 9490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:25.934367 9490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:25.934506 9490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:25.934628 9490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260812 11:28:25.970427 9490 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:45647
I20260812 11:28:25.970487 9705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:45647 every 8 connection(s)
I20260812 11:28:25.972023 9490 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:28:25.977567 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:25.975294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:36191)
0812 11:28:25.976472 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:25.976726 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":534,"threads_started":1}
W20260812 11:28:25.978252 9706 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:42741 (0 consecutive failures): Network error: Failed to ping master at 127.12.110.254:42741: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
I20260812 11:28:25.979132 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:25.978801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:36867)
0812 11:28:25.978961 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:25.979032 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 11:28:25.979964 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:25.979639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:49543)
0812 11:28:25.979780 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:25.979861 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260812 11:28:25.980225 9706 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:28:25.981806 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 9490
I20260812 11:28:25.982484 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:38413
--local_ip_for_outbound_sockets=127.12.110.195
--tserver_master_addrs=127.12.110.254:42741
--webserver_port=42089
--webserver_interface=127.12.110.195
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:26.331925 9715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:26.332252 9715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:26.332439 9715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:26.339545 9715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:26.339697 9715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:26.339877 9715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:28:26.349413 9715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:38413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=42089
--tserver_master_addrs=127.12.110.254:42741
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:26.350728 9715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:26.352007 9715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:26.365408 9728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:26.365248 9724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:26.365403 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:26.366263 9715 server_base.cc:1034] running on GCE node
I20260812 11:28:26.366912 9715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:26.367913 9715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:26.369139 9715 hybrid_clock.cc:648] HybridClock initialized: now 1786534106369094 us; error 38 us; skew 500 ppm
I20260812 11:28:26.369424 9715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:26 dist-test-slave-h6n0 krb5kdc[7042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534106, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:26.372772 9715 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:28:26.374275 9715 webserver.cc:460] Webserver started at http://127.12.110.195:42089/ using document root <none> and password file <none>
I20260812 11:28:26.375039 9715 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:26.375204 9715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:26.381320 9715 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.006s sys 0.000s
I20260812 11:28:26.384176 9739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:26.385917 9715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260812 11:28:26.386089 9715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "a9f015c5d9844f7c82d84ccefa26e9a9"
format_stamp: "Formatted at 2026-08-12 11:27:40 on dist-test-slave-h6n0"
server_key: "3a5cbf03cf6126ae59b0a5f434aaeba9"
server_key_iv: "782c4b9d7f4c7aa28eb92f49ba6ac1f7"
server_key_version: "encryptionkey@0"
I20260812 11:28:26.387575 9715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:26.405287 9715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:26.409869 9715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:26.410147 9715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:26.411262 9715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:26.412549 9715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:26.412643 9715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:26.412791 9715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:26.412890 9715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:26.447050 9715 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:38413
I20260812 11:28:26.447096 9942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:38413 every 8 connection(s)
I20260812 11:28:26.448541 9715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:28:26.449684 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 9715
I20260812 11:28:26.453405 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.451637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:50493)
0812 11:28:26.452370 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:26.452599 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":289,"threads_started":1}
W20260812 11:28:26.454058 9943 heartbeater.cc:644] Failed to heartbeat to 127.12.110.254:42741 (0 consecutive failures): Network error: Failed to ping master at 127.12.110.254:42741: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
I20260812 11:28:26.455470 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.454969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:53653)
0812 11:28:26.455180 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:26.455306 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260812 11:28:26.456660 9950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.456202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:43411)
0812 11:28:26.456412 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:26.456516 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260812 11:28:26.457094 9943 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260812 11:28:26.491775 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.490963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:58541)
0812 11:28:26.491412 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:26.491532 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":209,"threads_started":1}
I20260812 11:28:26.752265 9955 raft_consensus.cc:482] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:26.752650 9955 raft_consensus.cc:504] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:26.754843 9955 leader_election.cc:288] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
Aug 12 11:28:26 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534105, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
Aug 12 11:28:26 dist-test-slave-h6n0 krb5kdc[7042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534105, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:28:26.780144 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.755229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:45647 (local address 127.12.110.193:52703)
0812 11:28:26.755598 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:26.755646 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:26.756378 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:26.761177 (+ 4799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:26.761184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:26.762069 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:26.762870 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:26.762892 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.764780 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.764784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:26.766242 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:26.766255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.767390 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.767398 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:26.767568 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:26.768491 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:26.768510 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:26.771258 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:26.775403 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:26.775412 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:26.775426 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:26.776254 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:26.776802 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:26.776808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:26.776811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:26.777094 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:26.778210 (+ 1116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:26.778214 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:26.778693 (+ 479us) client_negotiation.cc:769] Sending connection context
0812 11:28:26.779061 (+ 368us) client_negotiation.cc:240] Negotiation successful
0812 11:28:26.779755 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":382,"spinlock_wait_cycles":37376}
I20260812 11:28:26.780174 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.755516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:52703 (local address 127.12.110.194:45647)
0812 11:28:26.757361 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:26.757383 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:26.757436 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:28:26.760113 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:26.760129 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:26.760735 (+ 606us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:26.761328 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:26.763102 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.764558 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.766404 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.767158 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.767394 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:26.771622 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:26.771635 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:26.771653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:26.771707 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:26.775213 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:26.776432 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:26.776440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:26.776448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:26.776656 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:26.777230 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:26.777236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:26.777240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:26.777975 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:26.778163 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:26.779312 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0812 11:28:26.779748 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1597,"thread_start_us":1397,"threads_started":1}
I20260812 11:28:26.780174 9961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.755468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:48327 (local address 127.12.110.195:38413)
0812 11:28:26.757722 (+ 2254us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:26.757738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:26.758306 (+ 568us) server_negotiation.cc:407] Connection header received
0812 11:28:26.760217 (+ 1911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:26.760226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:26.760616 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:26.760958 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:26.763069 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.764248 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.766408 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.767103 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.767301 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:26.772012 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:26.772027 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:26.772038 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:26.772076 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:26.775460 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:26.776432 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:26.776440 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:26.776448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:26.776659 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:26.777230 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:26.777236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:26.777240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:26.777976 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:26.778165 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:26.779312 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0812 11:28:26.779733 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2057,"thread_start_us":1897,"threads_started":1}
I20260812 11:28:26.780153 9959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.755286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:38413 (local address 127.12.110.193:48327)
0812 11:28:26.758168 (+ 2882us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:26.758184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:26.758370 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:26.761081 (+ 2711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:26.761096 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:26.762067 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:26.762869 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:26.762892 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.764445 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.764449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:26.766236 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:26.766254 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:26.767303 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:26.767310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:26.767566 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:26.768301 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:26.768341 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:26.771658 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:26.775588 (+ 3930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:26.775593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:26.775597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:26.776254 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:26.776802 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:26.776807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:26.776811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:26.777094 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:26.778198 (+ 1104us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:26.778207 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:26.778693 (+ 486us) client_negotiation.cc:769] Sending connection context
0812 11:28:26.779060 (+ 367us) client_negotiation.cc:240] Negotiation successful
0812 11:28:26.779631 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2592,"mutex_wait_us":131,"spinlock_wait_cycles":11904,"thread_start_us":2394,"threads_started":1}
W20260812 11:28:26.783957 9282 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
W20260812 11:28:26.784775 9285 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:26.784891 9285 leader_election.cc:302] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6523020d0bf4ddfbe9ad4c3befcb86d; no voters: 42b3d5d1573446f683cf83e49f8d85d4, a9f015c5d9844f7c82d84ccefa26e9a9
I20260812 11:28:26.785259 9955 raft_consensus.cc:2726] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:28:26.981704 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:26.980796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:40387)
0812 11:28:26.981309 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:26.981456 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":267,"threads_started":1}
I20260812 11:28:27.458313 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:27.457562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:51999)
0812 11:28:27.457998 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:27.458110 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":214,"threads_started":1}
I20260812 11:28:27.493448 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:27.492720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:52937)
0812 11:28:27.493207 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:27.493286 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":290,"threads_started":1}
I20260812 11:28:27.983371 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:27.982718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:35083)
0812 11:28:27.983138 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:27.983219 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":225,"threads_started":1}
I20260812 11:28:28.460408 9971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:28.459427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:50391)
0812 11:28:28.460098 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:28.460194 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":298,"threads_started":1}
I20260812 11:28:28.495302 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:28.494449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:46081)
0812 11:28:28.495039 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:28.495125 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":305,"threads_started":1}
I20260812 11:28:28.958488 9975 raft_consensus.cc:482] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:28.958761 9975 raft_consensus.cc:504] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:28.959408 9975 leader_election.cc:288] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
W20260812 11:28:28.960614 9285 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
W20260812 11:28:28.960803 9282 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:28.961010 9282 leader_election.cc:302] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6523020d0bf4ddfbe9ad4c3befcb86d; no voters: 42b3d5d1573446f683cf83e49f8d85d4, a9f015c5d9844f7c82d84ccefa26e9a9
I20260812 11:28:28.961437 9975 raft_consensus.cc:2726] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:28:28.985451 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:28.984258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:54751)
0812 11:28:28.984963 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:28.985117 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":278,"threads_started":1}
I20260812 11:28:29.462755 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:29.461521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:49081)
0812 11:28:29.462377 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:29.462512 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":299,"threads_started":1}
I20260812 11:28:29.496891 9983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:29.496173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:58103)
0812 11:28:29.496634 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:29.496718 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
I20260812 11:28:29.987702 9985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:29.986762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:42505)
0812 11:28:29.987420 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:29.987508 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":339,"threads_started":1}
I20260812 11:28:30.464680 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:30.463828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:33947)
0812 11:28:30.464377 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:30.464476 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":307,"threads_started":1}
I20260812 11:28:30.498878 9988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:30.497997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:55851)
0812 11:28:30.498582 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:30.498677 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":261,"threads_started":1}
I20260812 11:28:30.989529 9991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:30.988761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:44363)
0812 11:28:30.989254 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:30.989345 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260812 11:28:31.467073 9992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:31.466066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:43407)
0812 11:28:31.466822 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:31.466911 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":420,"threads_started":1}
I20260812 11:28:31.500931 9993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:31.500008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:56129)
0812 11:28:31.500575 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:31.500715 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":300,"threads_started":1}
I20260812 11:28:31.991268 9995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:31.990380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:34053)
0812 11:28:31.990910 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:31.990995 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":320,"threads_started":1}
I20260812 11:28:32.227507 10000 raft_consensus.cc:482] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:32.227711 10000 raft_consensus.cc:504] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:32.228367 10000 leader_election.cc:288] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
W20260812 11:28:32.229198 9282 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
W20260812 11:28:32.229377 9285 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:32.229535 9285 leader_election.cc:302] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6523020d0bf4ddfbe9ad4c3befcb86d; no voters: 42b3d5d1573446f683cf83e49f8d85d4, a9f015c5d9844f7c82d84ccefa26e9a9
I20260812 11:28:32.229810 10000 raft_consensus.cc:2726] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:28:32.468677 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:32.467897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:58937)
0812 11:28:32.468390 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:32.468472 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":283,"threads_started":1}
I20260812 11:28:32.502678 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:32.501842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:33603)
0812 11:28:32.502304 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:32.502407 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":238,"threads_started":1}
I20260812 11:28:32.993242 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:32.992459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:59339)
0812 11:28:32.992949 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:32.993053 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":220,"threads_started":1}
I20260812 11:28:33.470470 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:33.469576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:41367)
0812 11:28:33.470175 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:33.470256 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":260,"threads_started":1}
I20260812 11:28:33.504272 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:33.503560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:37577)
0812 11:28:33.504030 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:33.504107 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260812 11:28:33.995056 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:33.994275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:51223)
0812 11:28:33.994784 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:33.994865 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":292,"threads_started":1}
I20260812 11:28:34.472168 10013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:34.471320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:40355)
0812 11:28:34.471815 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:34.471915 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":251,"threads_started":1}
I20260812 11:28:34.505793 10015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:34.505078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:33843)
0812 11:28:34.505540 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:34.505617 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
I20260812 11:28:34.996624 10016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:34.995894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:59481)
0812 11:28:34.996362 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:34.996442 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":217,"threads_started":1}
I20260812 11:28:35.473830 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:35.473066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:47011)
0812 11:28:35.473586 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:35.473674 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":248,"threads_started":1}
I20260812 11:28:35.507591 10019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:35.506891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:57799)
0812 11:28:35.507347 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:35.507429 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":236,"threads_started":1}
I20260812 11:28:35.998245 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:35.997546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:42211)
0812 11:28:35.997972 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:35.998062 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260812 11:28:36.229854 10022 raft_consensus.cc:482] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:36.230067 10022 raft_consensus.cc:504] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:36.230904 10022 leader_election.cc:288] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
W20260812 11:28:36.231926 9282 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
W20260812 11:28:36.232108 9285 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:36.232242 9285 leader_election.cc:302] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6523020d0bf4ddfbe9ad4c3befcb86d; no voters: 42b3d5d1573446f683cf83e49f8d85d4, a9f015c5d9844f7c82d84ccefa26e9a9
I20260812 11:28:36.232476 10022 raft_consensus.cc:2726] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:28:36.475561 10024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:36.474797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:41919)
0812 11:28:36.475280 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:36.475368 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":263,"threads_started":1}
I20260812 11:28:36.509491 10026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:36.508459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:39261)
0812 11:28:36.509184 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:36.509281 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":467,"threads_started":1}
I20260812 11:28:37.000435 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:36.999541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:45205)
0812 11:28:37.000156 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:37.000264 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":326,"threads_started":1}
I20260812 11:28:37.477397 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:37.476560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:53193)
0812 11:28:37.477152 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:37.477232 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":325,"threads_started":1}
I20260812 11:28:37.511014 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:37.510326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:44723)
0812 11:28:37.510783 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:37.510860 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":267,"threads_started":1}
I20260812 11:28:38.002146 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:38.001422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:60797)
0812 11:28:38.001886 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:38.001971 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
I20260812 11:28:38.479257 10034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:38.478422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:52895)
0812 11:28:38.478996 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:38.479082 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":322,"threads_started":1}
I20260812 11:28:38.513271 10036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:38.512305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:40207)
0812 11:28:38.512865 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:38.512975 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":318,"threads_started":1}
I20260812 11:28:39.004230 10037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:39.003350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:46143)
0812 11:28:39.003922 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:39.004035 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":249,"threads_started":1}
I20260812 11:28:39.480849 10038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:39.480116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:36555)
0812 11:28:39.480557 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:39.480670 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":227,"threads_started":1}
I20260812 11:28:39.515079 10039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:39.514216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:42191)
0812 11:28:39.514788 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:39.514890 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":325,"threads_started":1}
I20260812 11:28:40.006925 10040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:40.005822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:47777)
0812 11:28:40.006446 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:40.006716 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":389,"threads_started":1}
I20260812 11:28:40.482775 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:40.481779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:51743)
0812 11:28:40.482380 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:40.482492 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":351,"threads_started":1}
I20260812 11:28:40.517470 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:40.516612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:38143)
0812 11:28:40.517191 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:40.517282 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":303,"threads_started":1}
I20260812 11:28:40.656490 10045 raft_consensus.cc:482] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:40.656802 10045 raft_consensus.cc:504] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6523020d0bf4ddfbe9ad4c3befcb86d" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 37869 } } peers { permanent_uuid: "a9f015c5d9844f7c82d84ccefa26e9a9" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 38413 } } peers { permanent_uuid: "42b3d5d1573446f683cf83e49f8d85d4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 45647 } }
I20260812 11:28:40.657617 10045 leader_election.cc:288] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413), 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647)
W20260812 11:28:40.658731 9282 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9f015c5d9844f7c82d84ccefa26e9a9 (127.12.110.195:38413): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
W20260812 11:28:40.658927 9285 leader_election.cc:341] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42b3d5d1573446f683cf83e49f8d85d4 (127.12.110.194:45647): Not found: Tablet not found: d0e273ceeefc49cfa7d97df14dc2ccca
I20260812 11:28:40.659051 9285 leader_election.cc:302] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6523020d0bf4ddfbe9ad4c3befcb86d; no voters: 42b3d5d1573446f683cf83e49f8d85d4, a9f015c5d9844f7c82d84ccefa26e9a9
I20260812 11:28:40.659281 10045 raft_consensus.cc:2726] T d0e273ceeefc49cfa7d97df14dc2ccca P c6523020d0bf4ddfbe9ad4c3befcb86d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260812 11:28:41.008903 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:41.008165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.194:57953)
0812 11:28:41.008617 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:41.008744 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
I20260812 11:28:41.452970 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 9176
I20260812 11:28:41.472154 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:41.454436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.12.110.193:37869)
0812 11:28:41.455690 (+ 1254us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:41.455697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:41.455722 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:28:41.455785 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:41.455791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:41.456003 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:41.456139 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:41.457707 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:41.458734 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:41.460056 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:41.460626 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:41.460734 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:41.463996 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:41.464008 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:41.464023 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:41.464083 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:41.468395 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:41.469347 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:41.469359 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:41.469361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:41.469581 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:41.470007 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:41.470011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:41.470014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:41.470460 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:41.470853 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:41.471340 (+ 487us) server_negotiation.cc:299] Negotiation successful
0812 11:28:41.471799 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1113,"thread_start_us":1027,"threads_started":1}
I20260812 11:28:41.473783 9478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:41.484989 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:41.484124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.195:55145)
0812 11:28:41.484716 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:41.484816 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":338,"threads_started":1}
I20260812 11:28:41.567391 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:41.566249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42741 (local address 127.12.110.193:53067)
0812 11:28:41.566891 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:41.567060 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.254:42741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":373,"threads_started":1}
I20260812 11:28:41.572575 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 9176
I20260812 11:28:41.602931 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 9490
I20260812 11:28:41.618445 10052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:41.603750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58140 (local address 127.12.110.194:45647)
0812 11:28:41.604443 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:41.604451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:41.604479 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:28:41.604697 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:41.604705 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:41.604812 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:41.605019 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:41.606764 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:41.607650 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:41.609064 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:41.609519 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:41.609665 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:41.612438 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:41.612443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:41.612446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:41.612476 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:41.615435 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:41.616073 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:41.616086 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:41.616088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:41.616212 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:41.616588 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:41.616593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:41.616597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:41.616868 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:41.617149 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:41.617763 (+ 614us) server_negotiation.cc:299] Negotiation successful
0812 11:28:41.618048 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":352,"threads_started":1}
I20260812 11:28:41.619376 9701 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:41.708029 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 9490
I20260812 11:28:41.735486 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 9715
I20260812 11:28:41.750785 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:41.736385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.12.110.195:38413)
0812 11:28:41.736954 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:41.736958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:41.736975 (+ 17us) server_negotiation.cc:407] Connection header received
0812 11:28:41.737035 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:41.737039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:41.737107 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:41.737233 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:41.738766 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:41.739574 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:41.740833 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:41.741306 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:41.741447 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:41.744571 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:41.744577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:41.744580 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:41.744612 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:41.747316 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:41.748223 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:41.748232 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:41.748239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:41.748425 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:41.748903 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:41.748911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:41.748915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:41.749268 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:41.749625 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:41.749948 (+ 323us) server_negotiation.cc:299] Negotiation successful
0812 11:28:41.750251 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":308,"threads_started":1}
I20260812 11:28:41.751686 9937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:41.837249 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 9715
2026-08-12T11:28:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65436 ms)
[----------] 30 tests from SecurityITest (318957 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:28:41 dist-test-slave-h6n0 krb5kdc[10060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:28:41 dist-test-slave-h6n0 krb5kdc[10060](info): set up 2 sockets
Aug 12 11:28:41 dist-test-slave-h6n0 krb5kdc[10060](info): commencing operation
krb5kdc: starting...
W20260812 11:28:43.274942 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.381s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:28:43 dist-test-slave-h6n0 krb5kdc[10060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:28:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:28:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:28:43.622468 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37521
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:43.981002 10083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:43.981288 10083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:43.981383 10083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:43.988737 10083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:28:43.988858 10083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:43.988909 10083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:43.988973 10083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:28:43.989022 10083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:28:43.998786 10083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:43.999989 10083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:44.001600 10083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:44.013446 10096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:44.013451 10097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:44.013479 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:44.014904 10083 server_base.cc:1034] running on GCE node
I20260812 11:28:44.015710 10083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:44.017123 10083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:44.018368 10083 hybrid_clock.cc:648] HybridClock initialized: now 1786534124018314 us; error 32 us; skew 500 ppm
I20260812 11:28:44.018784 10083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:44 dist-test-slave-h6n0 krb5kdc[10060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:44.022049 10083 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:28:44.023671 10083 webserver.cc:460] Webserver started at http://127.12.110.254:39887/ using document root <none> and password file <none>
I20260812 11:28:44.024307 10083 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:44.024416 10083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:44.024662 10083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:44.027915 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "7dc83aaf5fd34ababf287a3b7a2858fb"
format_stamp: "Formatted at 2026-08-12 11:28:44 on dist-test-slave-h6n0"
server_key: "6457871c5a4755c5b4e16fe494c43a96"
server_key_iv: "acff257b6f15784dda072830bdde50fc"
server_key_version: "encryptionkey@0"
I20260812 11:28:44.028626 10083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "7dc83aaf5fd34ababf287a3b7a2858fb"
format_stamp: "Formatted at 2026-08-12 11:28:44 on dist-test-slave-h6n0"
server_key: "6457871c5a4755c5b4e16fe494c43a96"
server_key_iv: "acff257b6f15784dda072830bdde50fc"
server_key_version: "encryptionkey@0"
I20260812 11:28:44.033638 10083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 11:28:44.037016 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:44.038309 10083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:28:44.038566 10083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "7dc83aaf5fd34ababf287a3b7a2858fb"
format_stamp: "Formatted at 2026-08-12 11:28:44 on dist-test-slave-h6n0"
server_key: "6457871c5a4755c5b4e16fe494c43a96"
server_key_iv: "acff257b6f15784dda072830bdde50fc"
server_key_version: "encryptionkey@0"
I20260812 11:28:44.038792 10083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:44.064590 10083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:44.069119 10083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:44.069423 10083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:44.089581 10083 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37521
I20260812 11:28:44.089612 10208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37521 every 8 connection(s)
I20260812 11:28:44.091014 10083 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:28:44.094853 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:44.098146 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 10083
I20260812 11:28:44.098379 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:28:44.099274 12731 external_mini_cluster.cc:1428] Setting key 4e7dad36706d7fef9ecb45cebeee10bc
I20260812 11:28:44.109437 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb: Bootstrap starting.
I20260812 11:28:44.113131 10209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:44.114131 10209 log.cc:824] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb: Log is configured to *not* fsync() on all Append() calls
Aug 12 11:28:44 dist-test-slave-h6n0 krb5kdc[10060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:44.117111 10209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb: No bootstrap required, opened a new log
I20260812 11:28:44.126487 10213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:44.101756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51490 (local address 127.12.110.254:37521)
0812 11:28:44.102581 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:44.102597 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:44.102649 (+ 52us) server_negotiation.cc:407] Connection header received
0812 11:28:44.104803 (+ 2154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:44.104841 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:44.105350 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:44.105773 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:44.107194 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:44.109252 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:44.110815 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:44.111887 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:44.112093 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:44.117360 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:44.117371 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:44.117388 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:44.117442 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:44.121770 (+ 4328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:44.122560 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:44.122568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:44.122575 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:44.122813 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:44.123216 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:44.123222 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:44.123231 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:44.123934 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:44.124267 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:44.124677 (+ 410us) server_negotiation.cc:299] Negotiation successful
0812 11:28:44.125130 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":336,"threads_started":1}
I20260812 11:28:44.128433 10209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37521 } }
I20260812 11:28:44.128823 10209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:44.128952 10209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dc83aaf5fd34ababf287a3b7a2858fb, State: Initialized, Role: FOLLOWER
I20260812 11:28:44.129613 10209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37521 } }
I20260812 11:28:44.129870 10209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:28:44.129998 10209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:28:44.130179 10209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:44.132359 10209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37521 } }
I20260812 11:28:44.132833 10209 leader_election.cc:302] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dc83aaf5fd34ababf287a3b7a2858fb; no voters:
I20260812 11:28:44.133490 10209 leader_election.cc:288] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:28:44.133615 10215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:44.134455 10215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [term 1 LEADER]: Becoming Leader. State: Replica: 7dc83aaf5fd34ababf287a3b7a2858fb, State: Running, Role: LEADER
I20260812 11:28:44.135075 10215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37521 } }
I20260812 11:28:44.135691 10209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:28:44.137809 10217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37521 } } }
I20260812 11:28:44.138254 10217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:44.139019 10218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dc83aaf5fd34ababf287a3b7a2858fb. Latest consensus state: current_term: 1 leader_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dc83aaf5fd34ababf287a3b7a2858fb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37521 } } }
I20260812 11:28:44.139416 10218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:44.140681 10226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:28:44.145248 10226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:28:44.153825 10226 catalog_manager.cc:1347] Generated new cluster ID: 36edf33cccf840ae94c7820d4b6d3ea0
I20260812 11:28:44.153947 10226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:28:44.185545 10226 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:28:44.186965 10226 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:28:44.200759 10226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb: Generated new TSK 0
I20260812 11:28:44.201661 10226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:28:44.279587 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37521
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:44.655411 10251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:44.655681 10251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:44.655898 10251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:44.663094 10251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:44.663200 10251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:44.663432 10251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:28:44.673193 10251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:44.674480 10251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:44.675968 10251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:44.692027 10257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:44.692080 10258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:44.692121 10263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:44.692937 10251 server_base.cc:1034] running on GCE node
I20260812 11:28:44.693599 10251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:44.694603 10251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:44.695832 10251 hybrid_clock.cc:648] HybridClock initialized: now 1786534124695773 us; error 47 us; skew 500 ppm
I20260812 11:28:44.696211 10251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:44 dist-test-slave-h6n0 krb5kdc[10060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:44.699898 10251 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:28:44.701870 10251 webserver.cc:460] Webserver started at http://127.12.110.193:44253/ using document root <none> and password file <none>
I20260812 11:28:44.702631 10251 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:44.702782 10251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:44.703208 10251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:44.706177 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "897afe093892439c9c4a09ce643d9622"
format_stamp: "Formatted at 2026-08-12 11:28:44 on dist-test-slave-h6n0"
server_key: "ddc3ba07cefb81f7e64911a7ac6dd804"
server_key_iv: "e05f7b6cd35a300eed089d5c11c670a2"
server_key_version: "encryptionkey@0"
I20260812 11:28:44.707048 10251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "897afe093892439c9c4a09ce643d9622"
format_stamp: "Formatted at 2026-08-12 11:28:44 on dist-test-slave-h6n0"
server_key: "ddc3ba07cefb81f7e64911a7ac6dd804"
server_key_iv: "e05f7b6cd35a300eed089d5c11c670a2"
server_key_version: "encryptionkey@0"
I20260812 11:28:44.712203 10251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:28:44.716574 10278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:44.718436 10251 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260812 11:28:44.718677 10251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "897afe093892439c9c4a09ce643d9622"
format_stamp: "Formatted at 2026-08-12 11:28:44 on dist-test-slave-h6n0"
server_key: "ddc3ba07cefb81f7e64911a7ac6dd804"
server_key_iv: "e05f7b6cd35a300eed089d5c11c670a2"
server_key_version: "encryptionkey@0"
I20260812 11:28:44.718927 10251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:44.746039 10251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:44.751821 10251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:44.752378 10251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:44.753842 10251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:44.755628 10251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:44.755779 10251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:44.755954 10251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:44.756034 10251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:44.792882 10251 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:39039
I20260812 11:28:44.792853 10472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:39039 every 8 connection(s)
I20260812 11:28:44.794245 10251 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:28:44.795879 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 10251
I20260812 11:28:44.796114 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:28:44.797017 12731 external_mini_cluster.cc:1428] Setting key f7e9902de4d1abddcc633b8d8647f22e
Aug 12 11:28:44 dist-test-slave-h6n0 krb5kdc[10060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534124, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:44.818081 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:44.797170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:33319 (local address 127.12.110.254:37521)
0812 11:28:44.798032 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:44.798041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:44.798815 (+ 774us) server_negotiation.cc:407] Connection header received
0812 11:28:44.801695 (+ 2880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:44.801700 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:44.801763 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:44.801887 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:44.803978 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:44.804690 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:44.807248 (+ 2558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:44.807639 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:44.807737 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:44.812302 (+ 4565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:44.812307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:44.812310 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:44.812342 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:44.815007 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:44.815741 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:44.815746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:44.815748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:44.815868 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:44.816330 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:44.816334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:44.816336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:44.816599 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:44.816772 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:44.817479 (+ 707us) server_negotiation.cc:299] Negotiation successful
0812 11:28:44.817725 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":578,"threads_started":1}
I20260812 11:28:44.819097 10479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:44.797784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37521 (local address 127.12.110.193:33319)
0812 11:28:44.798667 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:44.798713 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:44.801396 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:44.802166 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:44.802178 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:44.803012 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:44.803785 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:44.803812 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:44.804860 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:44.804868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:44.807077 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:44.807099 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:44.807877 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:44.807888 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:44.808245 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:44.809127 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:44.809163 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:44.812064 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:44.815152 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:44.815161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:44.815176 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:44.815603 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:44.815992 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:44.815997 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:44.816001 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:44.816195 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:44.816751 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:44.816758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:44.817089 (+ 331us) client_negotiation.cc:769] Sending connection context
0812 11:28:44.817356 (+ 267us) client_negotiation.cc:240] Negotiation successful
0812 11:28:44.817866 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":346,"threads_started":1}
I20260812 11:28:44.820721 10473 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37521
I20260812 11:28:44.821029 10473 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:44.821744 10473 heartbeater.cc:511] Master 127.12.110.254:37521 requested a full tablet report, sending...
I20260812 11:28:44.824160 10148 ts_manager.cc:194] Registered new tserver with Master: 897afe093892439c9c4a09ce643d9622 (127.12.110.193:39039)
I20260812 11:28:44.825631 10148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:33319
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:28:44.874289 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37521
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:45.215204 10483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:45.215533 10483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:45.215698 10483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:45.223001 10483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:45.223145 10483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:45.223330 10483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:28:45.233273 10483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:45.234586 10483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:45.235859 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:45.248570 10500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:45.248613 10494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:45.248574 10497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:45.249799 10483 server_base.cc:1034] running on GCE node
I20260812 11:28:45.250411 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:45.251408 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:45.252712 10483 hybrid_clock.cc:648] HybridClock initialized: now 1786534125252618 us; error 86 us; skew 500 ppm
I20260812 11:28:45.253115 10483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:45 dist-test-slave-h6n0 krb5kdc[10060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:45.256664 10483 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:28:45.258368 10483 webserver.cc:460] Webserver started at http://127.12.110.194:40763/ using document root <none> and password file <none>
I20260812 11:28:45.259153 10483 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:45.259310 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:45.259644 10483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:45.262624 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "eeedfd38d00245da90850175d8df1dc3"
format_stamp: "Formatted at 2026-08-12 11:28:45 on dist-test-slave-h6n0"
server_key: "a8ae39484022387e5ed3843cf90d16c3"
server_key_iv: "2239839138fc9d014efa5aaa16f9b001"
server_key_version: "encryptionkey@0"
I20260812 11:28:45.263414 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "eeedfd38d00245da90850175d8df1dc3"
format_stamp: "Formatted at 2026-08-12 11:28:45 on dist-test-slave-h6n0"
server_key: "a8ae39484022387e5ed3843cf90d16c3"
server_key_iv: "2239839138fc9d014efa5aaa16f9b001"
server_key_version: "encryptionkey@0"
I20260812 11:28:45.268608 10483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:28:45.272555 10509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:45.273976 10483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:28:45.274199 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "eeedfd38d00245da90850175d8df1dc3"
format_stamp: "Formatted at 2026-08-12 11:28:45 on dist-test-slave-h6n0"
server_key: "a8ae39484022387e5ed3843cf90d16c3"
server_key_iv: "2239839138fc9d014efa5aaa16f9b001"
server_key_version: "encryptionkey@0"
I20260812 11:28:45.274443 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:45.300025 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:45.304611 10483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:45.304950 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:45.306015 10483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:45.309052 10483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:45.309139 10483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:45.309232 10483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:45.309269 10483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:45.345404 10483 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:36157
I20260812 11:28:45.345453 10700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:36157 every 8 connection(s)
I20260812 11:28:45.346760 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:28:45.347137 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 10483
I20260812 11:28:45.347350 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:28:45.348021 12731 external_mini_cluster.cc:1428] Setting key 828413626a08125474f9ae16d3273ce9
Aug 12 11:28:45 dist-test-slave-h6n0 krb5kdc[10060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:45.366281 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.349470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:39003 (local address 127.12.110.254:37521)
0812 11:28:45.349967 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:45.349973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:45.351062 (+ 1089us) server_negotiation.cc:407] Connection header received
0812 11:28:45.352864 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:45.352870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:45.352955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:45.353085 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:45.354507 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.355232 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.356717 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.357153 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.357288 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.360725 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:45.360730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:45.360733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:45.360764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:45.363376 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:45.364060 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:45.364068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:45.364074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:45.364208 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:45.364641 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:45.364647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:45.364650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:45.364911 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:45.365070 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:45.365686 (+ 616us) server_negotiation.cc:299] Negotiation successful
0812 11:28:45.365903 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
I20260812 11:28:45.367166 10711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.350120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37521 (local address 127.12.110.194:39003)
0812 11:28:45.350916 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:45.350953 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:45.352614 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:45.353309 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:45.353317 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:45.353813 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:45.354340 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.354358 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.355384 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.355390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.356561 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.356574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.357379 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.357386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.357604 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.358313 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:45.358338 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:45.360502 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:45.363512 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:45.363520 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:45.363533 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:45.363941 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:45.364326 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:45.364331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:45.364334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:45.364525 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:45.365042 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:45.365049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:45.365338 (+ 289us) client_negotiation.cc:769] Sending connection context
0812 11:28:45.365591 (+ 253us) client_negotiation.cc:240] Negotiation successful
0812 11:28:45.366073 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":375,"threads_started":1}
I20260812 11:28:45.368501 10702 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37521
I20260812 11:28:45.368814 10702 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:45.369705 10702 heartbeater.cc:511] Master 127.12.110.254:37521 requested a full tablet report, sending...
I20260812 11:28:45.371274 10148 ts_manager.cc:194] Registered new tserver with Master: eeedfd38d00245da90850175d8df1dc3 (127.12.110.194:36157)
I20260812 11:28:45.372498 10148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:39003
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:28:45.419876 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37521
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:45.761842 10715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:45.762140 10715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:45.762311 10715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:45.769722 10715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:45.769821 10715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:45.770043 10715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:28:45.780062 10715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:45.781373 10715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:45.782707 10715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:45.795312 10724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:45.795866 10723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:45.796692 10730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:45.796916 10715 server_base.cc:1034] running on GCE node
I20260812 11:28:45.797667 10715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:45.798810 10715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:45.800084 10715 hybrid_clock.cc:648] HybridClock initialized: now 1786534125800028 us; error 56 us; skew 500 ppm
I20260812 11:28:45.800386 10715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:45 dist-test-slave-h6n0 krb5kdc[10060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:45.803668 10715 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:28:45.805142 10715 webserver.cc:460] Webserver started at http://127.12.110.195:46555/ using document root <none> and password file <none>
I20260812 11:28:45.805824 10715 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:45.805979 10715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:45.806290 10715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:45.809250 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "1cec117770e74088a5cadf5c1f467a1c"
format_stamp: "Formatted at 2026-08-12 11:28:45 on dist-test-slave-h6n0"
server_key: "7f16db1ed7260768087ea39ab2ba5f0f"
server_key_iv: "e547a5ce0d317f36dd336f45935baf81"
server_key_version: "encryptionkey@0"
I20260812 11:28:45.810022 10715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "1cec117770e74088a5cadf5c1f467a1c"
format_stamp: "Formatted at 2026-08-12 11:28:45 on dist-test-slave-h6n0"
server_key: "7f16db1ed7260768087ea39ab2ba5f0f"
server_key_iv: "e547a5ce0d317f36dd336f45935baf81"
server_key_version: "encryptionkey@0"
I20260812 11:28:45.815110 10715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:28:45.818359 10738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:45.819705 10715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:45.819900 10715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "1cec117770e74088a5cadf5c1f467a1c"
format_stamp: "Formatted at 2026-08-12 11:28:45 on dist-test-slave-h6n0"
server_key: "7f16db1ed7260768087ea39ab2ba5f0f"
server_key_iv: "e547a5ce0d317f36dd336f45935baf81"
server_key_version: "encryptionkey@0"
I20260812 11:28:45.820108 10715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:45.828691 10473 heartbeater.cc:503] Master 127.12.110.254:37521 was elected leader, sending a full tablet report...
I20260812 11:28:45.836844 10715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:45.841917 10715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:45.842204 10715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:45.843559 10715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:45.845172 10715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:45.845281 10715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:45.845418 10715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:45.845496 10715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:45.879037 10715 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:45561
I20260812 11:28:45.879060 10937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:45561 every 8 connection(s)
I20260812 11:28:45.880285 10715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:28:45.882110 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 10715
I20260812 11:28:45.882332 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:28:45.882987 12731 external_mini_cluster.cc:1428] Setting key 553cf134fd0c2d42225489b098907525
Aug 12 11:28:45 dist-test-slave-h6n0 krb5kdc[10060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534125, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:45.901827 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.882894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:34395 (local address 127.12.110.254:37521)
0812 11:28:45.883354 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:45.883359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:45.884549 (+ 1190us) server_negotiation.cc:407] Connection header received
0812 11:28:45.886490 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:45.886494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:45.886611 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:45.886708 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:45.888127 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.888893 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.890399 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.890954 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.891061 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.894598 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:45.894611 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:45.894614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:45.894647 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:45.898728 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:45.899520 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:45.899525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:45.899527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:45.899656 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:45.900073 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:45.900077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:45.900079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:45.900340 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:45.900673 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:45.901373 (+ 700us) server_negotiation.cc:299] Negotiation successful
0812 11:28:45.901543 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
I20260812 11:28:45.902860 10946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.883530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37521 (local address 127.12.110.195:34395)
0812 11:28:45.884386 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:45.884431 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:45.886254 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:45.886914 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:45.886921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:45.887409 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:45.887955 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.887971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.889061 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.889065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.890242 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.890255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.891186 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.891192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.891389 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.892063 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:45.892086 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:45.894327 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:45.898899 (+ 4572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:45.898908 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:45.898923 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:45.899381 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:45.899773 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:45.899776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:45.899778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:45.899969 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:45.900692 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:45.900698 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:45.901013 (+ 315us) client_negotiation.cc:769] Sending connection context
0812 11:28:45.901251 (+ 238us) client_negotiation.cc:240] Negotiation successful
0812 11:28:45.901610 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":357,"threads_started":1}
I20260812 11:28:45.904419 10938 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37521
I20260812 11:28:45.904735 10938 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:45.905467 10938 heartbeater.cc:511] Master 127.12.110.254:37521 requested a full tablet report, sending...
I20260812 11:28:45.907323 10148 ts_manager.cc:194] Registered new tserver with Master: 1cec117770e74088a5cadf5c1f467a1c (127.12.110.195:45561)
I20260812 11:28:45.908114 10148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:34395
I20260812 11:28:45.911393 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:28:45.929394 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.916275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.12.110.254:37521)
0812 11:28:45.916446 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:45.916450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:45.916979 (+ 529us) server_negotiation.cc:407] Connection header received
0812 11:28:45.917115 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:45.917118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:45.917208 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:45.917306 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:45.918947 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.919719 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.921063 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.921493 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.921590 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.924315 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:45.924323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:45.924328 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:45.924376 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:45.926971 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:45.927636 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:45.927640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:45.927642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:45.927759 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:45.928175 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:45.928178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:45.928180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:45.928496 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:45.928684 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:45.928982 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 11:28:45.929151 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:28:45.933487 10148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:28:45.940810 10148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:28:45.964121 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.953299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.12.110.194:36157)
0812 11:28:45.954306 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:45.954315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:45.954992 (+ 677us) server_negotiation.cc:407] Connection header received
0812 11:28:45.956033 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:45.956039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:45.956257 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:45.957339 (+ 1082us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:45.957581 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.959058 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.961722 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.962910 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.963096 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.963281 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:45.963597 (+ 316us) server_negotiation.cc:299] Negotiation successful
0812 11:28:45.963756 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":859,"threads_started":1}
I20260812 11:28:45.964219 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.953194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:36157 (local address 127.0.0.1:38574)
0812 11:28:45.954836 (+ 1642us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:45.954882 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:45.955083 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:45.956540 (+ 1457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:45.956547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:45.956579 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:45.957080 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.957100 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.959239 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.959245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.961561 (+ 2316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.961575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.963129 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.963137 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.963368 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.963405 (+ 37us) client_negotiation.cc:769] Sending connection context
0812 11:28:45.963557 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 11:28:45.963756 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1452,"mutex_wait_us":96,"thread_start_us":1287,"threads_started":1}
I20260812 11:28:45.964828 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.955133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:39039 (local address 127.0.0.1:52456)
0812 11:28:45.955731 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:45.955761 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:45.955884 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:45.956689 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:45.956696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:45.956739 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:45.957412 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.957429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.959725 (+ 2296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.959729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.962070 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.962085 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.963958 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.963964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.964141 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.964177 (+ 36us) client_negotiation.cc:769] Sending connection context
0812 11:28:45.964247 (+ 70us) client_negotiation.cc:240] Negotiation successful
0812 11:28:45.964366 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":24,"thread_start_us":278,"threads_started":1}
I20260812 11:28:45.965967 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.955417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.12.110.193:39039)
0812 11:28:45.956065 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:45.956073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:45.956102 (+ 29us) server_negotiation.cc:407] Connection header received
0812 11:28:45.956204 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:45.956213 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:45.956448 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:45.957634 (+ 1186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:45.957938 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.959532 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.962242 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.963690 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.965256 (+ 1566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.965405 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:45.965480 (+ 75us) server_negotiation.cc:299] Negotiation successful
0812 11:28:45.965632 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":302,"threads_started":1}
I20260812 11:28:45.966814 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.956027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.12.110.195:45561)
0812 11:28:45.956651 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:45.956661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:45.958408 (+ 1747us) server_negotiation.cc:407] Connection header received
0812 11:28:45.958619 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:45.958626 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:45.958822 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:45.959017 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:45.959671 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.961443 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.963873 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.965599 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.965777 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.965917 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:45.966386 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 11:28:45.966504 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":411,"threads_started":1}
I20260812 11:28:45.966851 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:45.956327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45561 (local address 127.0.0.1:43512)
0812 11:28:45.958261 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:45.958288 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:45.958431 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:45.959030 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:45.959034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:45.959061 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:45.959478 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.959488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.961587 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.961591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.963689 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:45.963703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:45.965968 (+ 2265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:45.965977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:45.966149 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:45.966219 (+ 70us) client_negotiation.cc:769] Sending connection context
0812 11:28:45.966312 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 11:28:45.966431 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1827,"mutex_wait_us":1,"thread_start_us":1689,"threads_started":1}
I20260812 11:28:45.968652 10592 tablet_service.cc:1467] Processing CreateTablet for tablet f563829fb2cf42cb80f602b3137d9653 (DEFAULT_TABLE table=test-table [id=46568ed36cdc45189feb434fbec3af9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:45.970036 10592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f563829fb2cf42cb80f602b3137d9653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:45.970090 10824 tablet_service.cc:1467] Processing CreateTablet for tablet f563829fb2cf42cb80f602b3137d9653 (DEFAULT_TABLE table=test-table [id=46568ed36cdc45189feb434fbec3af9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:45.970091 10356 tablet_service.cc:1467] Processing CreateTablet for tablet f563829fb2cf42cb80f602b3137d9653 (DEFAULT_TABLE table=test-table [id=46568ed36cdc45189feb434fbec3af9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:45.971513 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f563829fb2cf42cb80f602b3137d9653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:45.971531 10824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f563829fb2cf42cb80f602b3137d9653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:45.985370 10975 tablet_bootstrap.cc:492] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Bootstrap starting.
I20260812 11:28:45.988551 10976 tablet_bootstrap.cc:492] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Bootstrap starting.
I20260812 11:28:45.989096 10975 tablet_bootstrap.cc:654] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:45.989423 10977 tablet_bootstrap.cc:492] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Bootstrap starting.
I20260812 11:28:45.990592 10975 log.cc:824] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:45.993896 10975 tablet_bootstrap.cc:492] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: No bootstrap required, opened a new log
I20260812 11:28:45.994140 10977 tablet_bootstrap.cc:654] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:45.994140 10976 tablet_bootstrap.cc:654] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:45.994330 10975 ts_tablet_manager.cc:1397] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260812 11:28:45.995458 10976 log.cc:824] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:45.995482 10977 log.cc:824] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:45.998502 10976 tablet_bootstrap.cc:492] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: No bootstrap required, opened a new log
I20260812 11:28:45.998821 10976 ts_tablet_manager.cc:1397] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260812 11:28:45.999379 10977 tablet_bootstrap.cc:492] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: No bootstrap required, opened a new log
I20260812 11:28:45.999647 10977 ts_tablet_manager.cc:1397] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260812 11:28:46.003630 10975 raft_consensus.cc:348] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.004302 10975 raft_consensus.cc:374] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:46.004495 10975 raft_consensus.cc:729] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eeedfd38d00245da90850175d8df1dc3, State: Initialized, Role: FOLLOWER
I20260812 11:28:46.005129 10975 consensus_queue.cc:260] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.006268 10702 heartbeater.cc:503] Master 127.12.110.254:37521 was elected leader, sending a full tablet report...
I20260812 11:28:46.008241 10975 ts_tablet_manager.cc:1428] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260812 11:28:46.009804 10976 raft_consensus.cc:348] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.010267 10976 raft_consensus.cc:374] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:46.010381 10976 raft_consensus.cc:729] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cec117770e74088a5cadf5c1f467a1c, State: Initialized, Role: FOLLOWER
I20260812 11:28:46.010455 10977 raft_consensus.cc:348] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.010972 10977 raft_consensus.cc:374] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:46.011132 10977 raft_consensus.cc:729] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 897afe093892439c9c4a09ce643d9622, State: Initialized, Role: FOLLOWER
I20260812 11:28:46.011040 10976 consensus_queue.cc:260] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.012866 10938 heartbeater.cc:503] Master 127.12.110.254:37521 was elected leader, sending a full tablet report...
I20260812 11:28:46.013419 10976 ts_tablet_manager.cc:1428] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260812 11:28:46.013851 10977 consensus_queue.cc:260] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.015507 10977 ts_tablet_manager.cc:1428] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
W20260812 11:28:46.046944 10476 tablet.cc:2369] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:28:46.099256 10707 tablet.cc:2369] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:28:46.131954 10940 tablet.cc:2369] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:28:46.232360 10992 raft_consensus.cc:482] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:46.232735 10992 raft_consensus.cc:504] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.234670 10992 leader_election.cc:288] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eeedfd38d00245da90850175d8df1dc3 (127.12.110.194:36157), 1cec117770e74088a5cadf5c1f467a1c (127.12.110.195:45561)
I20260812 11:28:46.242219 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.235117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:51973 (local address 127.12.110.194:36157)
0812 11:28:46.235364 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:46.235369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:46.236092 (+ 723us) server_negotiation.cc:407] Connection header received
0812 11:28:46.236371 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:46.236381 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:46.236478 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:46.236549 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:46.237474 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.238419 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.240476 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.241239 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.241431 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.241505 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:46.241773 (+ 268us) server_negotiation.cc:299] Negotiation successful
0812 11:28:46.241898 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:28:46.242372 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.235116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:36157 (local address 127.12.110.193:51973)
0812 11:28:46.235932 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:46.235954 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:46.236122 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:46.236746 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:46.236748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:46.236775 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:46.237302 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:46.237313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.238575 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.238583 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:46.240308 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:46.240321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.241445 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.241452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:46.241613 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.241635 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:28:46.241746 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:28:46.241871 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"spinlock_wait_cycles":56704,"thread_start_us":451,"threads_started":1}
I20260812 11:28:46.242782 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.236450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:60281 (local address 127.12.110.195:45561)
0812 11:28:46.236673 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:46.236679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:46.236924 (+ 245us) server_negotiation.cc:407] Connection header received
0812 11:28:46.237241 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:46.237246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:46.237314 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:46.237422 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:46.238009 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.238937 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.240846 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.241702 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.241905 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.241993 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:46.242326 (+ 333us) server_negotiation.cc:299] Negotiation successful
0812 11:28:46.242465 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 11:28:46.243003 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.236056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:45561 (local address 127.12.110.193:60281)
0812 11:28:46.236794 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:46.236818 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:46.237074 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:46.237437 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:46.237442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:46.237467 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:46.237846 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:46.237855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.239095 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.239102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:46.240662 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:46.240675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.241953 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.241959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:46.242143 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.242168 (+ 25us) client_negotiation.cc:769] Sending connection context
0812 11:28:46.242285 (+ 117us) client_negotiation.cc:240] Negotiation successful
0812 11:28:46.242572 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"spinlock_wait_cycles":29440,"thread_start_us":307,"threads_started":1}
I20260812 11:28:46.243700 10627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f563829fb2cf42cb80f602b3137d9653" candidate_uuid: "897afe093892439c9c4a09ce643d9622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeedfd38d00245da90850175d8df1dc3" is_pre_election: true
I20260812 11:28:46.244120 10862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f563829fb2cf42cb80f602b3137d9653" candidate_uuid: "897afe093892439c9c4a09ce643d9622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cec117770e74088a5cadf5c1f467a1c" is_pre_election: true
I20260812 11:28:46.244362 10627 raft_consensus.cc:2454] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897afe093892439c9c4a09ce643d9622 in term 0.
I20260812 11:28:46.244666 10862 raft_consensus.cc:2454] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 897afe093892439c9c4a09ce643d9622 in term 0.
I20260812 11:28:46.245183 10279 leader_election.cc:302] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 897afe093892439c9c4a09ce643d9622, eeedfd38d00245da90850175d8df1dc3; no voters:
I20260812 11:28:46.245595 10992 raft_consensus.cc:2781] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:28:46.245764 10992 raft_consensus.cc:482] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:28:46.245900 10992 raft_consensus.cc:3037] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:46.248544 10992 raft_consensus.cc:504] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.249354 10992 leader_election.cc:288] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [CANDIDATE]: Term 1 election: Requested vote from peers eeedfd38d00245da90850175d8df1dc3 (127.12.110.194:36157), 1cec117770e74088a5cadf5c1f467a1c (127.12.110.195:45561)
I20260812 11:28:46.250229 10627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f563829fb2cf42cb80f602b3137d9653" candidate_uuid: "897afe093892439c9c4a09ce643d9622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eeedfd38d00245da90850175d8df1dc3"
I20260812 11:28:46.250303 10862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f563829fb2cf42cb80f602b3137d9653" candidate_uuid: "897afe093892439c9c4a09ce643d9622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cec117770e74088a5cadf5c1f467a1c"
I20260812 11:28:46.250430 10627 raft_consensus.cc:3037] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:46.250574 10862 raft_consensus.cc:3037] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:46.252756 10862 raft_consensus.cc:2454] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897afe093892439c9c4a09ce643d9622 in term 1.
I20260812 11:28:46.253129 10279 leader_election.cc:302] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cec117770e74088a5cadf5c1f467a1c, 897afe093892439c9c4a09ce643d9622; no voters:
I20260812 11:28:46.253118 10627 raft_consensus.cc:2454] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 897afe093892439c9c4a09ce643d9622 in term 1.
I20260812 11:28:46.253417 10992 raft_consensus.cc:2781] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:46.253942 10992 raft_consensus.cc:686] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 1 LEADER]: Becoming Leader. State: Replica: 897afe093892439c9c4a09ce643d9622, State: Running, Role: LEADER
I20260812 11:28:46.254505 10992 consensus_queue.cc:237] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } }
I20260812 11:28:46.258962 10147 catalog_manager.cc:5495] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 reported cstate change: term changed from 0 to 1, leader changed from <none> to 897afe093892439c9c4a09ce643d9622 (127.12.110.193). New cstate: current_term: 1 leader_uuid: "897afe093892439c9c4a09ce643d9622" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "897afe093892439c9c4a09ce643d9622" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 39039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 } health_report { overall_health: UNKNOWN } } }
I20260812 11:28:46.333257 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.327519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.12.110.193:39039)
0812 11:28:46.327755 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:46.327762 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:46.327870 (+ 108us) server_negotiation.cc:407] Connection header received
0812 11:28:46.328157 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:46.328161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:46.328243 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:46.328404 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:28:46.328797 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.329546 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.331020 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.331439 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.331591 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.332085 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:28:46.332666 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:28:46.332835 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:46.332892 (+ 57us) server_negotiation.cc:299] Negotiation successful
0812 11:28:46.332977 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:28:46.342518 10627 raft_consensus.cc:1261] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 1 FOLLOWER]: Refusing update from remote peer 897afe093892439c9c4a09ce643d9622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:46.342573 10862 raft_consensus.cc:1261] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 1 FOLLOWER]: Refusing update from remote peer 897afe093892439c9c4a09ce643d9622: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:46.343613 10996 consensus_queue.cc:1035] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cec117770e74088a5cadf5c1f467a1c" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 45561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:46.344033 10992 consensus_queue.cc:1035] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eeedfd38d00245da90850175d8df1dc3" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:46.380374 10146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:46.380729 10146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:46.384400 10146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7dc83aaf5fd34ababf287a3b7a2858fb: Sending DeleteTablet for 3 replicas of tablet f563829fb2cf42cb80f602b3137d9653
I20260812 11:28:46.385457 10592 tablet_service.cc:1514] Processing DeleteTablet for tablet f563829fb2cf42cb80f602b3137d9653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:46 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:38574
I20260812 11:28:46.385656 10824 tablet_service.cc:1514] Processing DeleteTablet for tablet f563829fb2cf42cb80f602b3137d9653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:46 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:43512
I20260812 11:28:46.385852 10356 tablet_service.cc:1514] Processing DeleteTablet for tablet f563829fb2cf42cb80f602b3137d9653 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:46 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:52456
I20260812 11:28:46.387866 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 10251
I20260812 11:28:46.417205 11024 tablet_replica.cc:330] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: stopping tablet replica
I20260812 11:28:46.417913 11024 raft_consensus.cc:2229] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:46.418702 11024 raft_consensus.cc:2258] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:46.425326 11024 ts_tablet_manager.cc:1905] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:46.426134 11020 tablet_replica.cc:330] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: stopping tablet replica
I20260812 11:28:46.426779 11027 tablet_replica.cc:330] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: stopping tablet replica
I20260812 11:28:46.427165 11027 raft_consensus.cc:2229] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:28:46.427830 11027 raft_consensus.cc:2258] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:46.429510 11020 raft_consensus.cc:2229] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:46.430078 11020 raft_consensus.cc:2258] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:46.434672 11020 ts_tablet_manager.cc:1905] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 11:28:46 dist-test-slave-h6n0 krb5kdc[10060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:28:46.436066 11027 ts_tablet_manager.cc:1905] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:46.444557 11027 ts_tablet_manager.cc:1918] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:46.444849 11027 log.cc:1196] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/f563829fb2cf42cb80f602b3137d9653
I20260812 11:28:46.446004 11024 ts_tablet_manager.cc:1918] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:46.446185 11024 log.cc:1196] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/f563829fb2cf42cb80f602b3137d9653
I20260812 11:28:46.446609 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.389692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.12.110.193:39039)
0812 11:28:46.389924 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:46.389933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:46.413498 (+ 23565us) server_negotiation.cc:407] Connection header received
0812 11:28:46.414110 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:46.414116 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:46.414218 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:46.414307 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:46.418994 (+ 4687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.420851 (+ 1857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.429153 (+ 8302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.429869 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.430017 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.437271 (+ 7254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:46.437292 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:46.437311 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:46.437409 (+ 98us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:46.442633 (+ 5224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:46.443623 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:46.443632 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:46.443636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:46.443821 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:46.444357 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:46.444363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:46.444368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:46.444893 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:46.445184 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:46.445839 (+ 655us) server_negotiation.cc:299] Negotiation successful
0812 11:28:46.446119 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260812 11:28:46.446734 11024 ts_tablet_manager.cc:1939] T f563829fb2cf42cb80f602b3137d9653 P 1cec117770e74088a5cadf5c1f467a1c: Deleting consensus metadata
I20260812 11:28:46.446089 11027 ts_tablet_manager.cc:1939] T f563829fb2cf42cb80f602b3137d9653 P 897afe093892439c9c4a09ce643d9622: Deleting consensus metadata
I20260812 11:28:46.447881 10468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:46.448688 10117 catalog_manager.cc:4841] TS 1cec117770e74088a5cadf5c1f467a1c (127.12.110.195:45561): tablet f563829fb2cf42cb80f602b3137d9653 (table test-table [id=46568ed36cdc45189feb434fbec3af9c]) successfully deleted
I20260812 11:28:46.457059 11020 ts_tablet_manager.cc:1918] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:46.457535 11020 log.cc:1196] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/f563829fb2cf42cb80f602b3137d9653
I20260812 11:28:46.458181 11020 ts_tablet_manager.cc:1939] T f563829fb2cf42cb80f602b3137d9653 P eeedfd38d00245da90850175d8df1dc3: Deleting consensus metadata
I20260812 11:28:46.459932 10117 catalog_manager.cc:4841] TS eeedfd38d00245da90850175d8df1dc3 (127.12.110.194:36157): tablet f563829fb2cf42cb80f602b3137d9653 (table test-table [id=46568ed36cdc45189feb434fbec3af9c]) successfully deleted
I20260812 11:28:46.555078 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 10251
I20260812 11:28:46.556233 10117 catalog_manager.cc:4841] TS 897afe093892439c9c4a09ce643d9622 (127.12.110.193:39039): tablet f563829fb2cf42cb80f602b3137d9653 (table test-table [id=46568ed36cdc45189feb434fbec3af9c]) successfully deleted
I20260812 11:28:46.582368 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 10483
Aug 12 11:28:46 dist-test-slave-h6n0 krb5kdc[10060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:28:46.602113 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.583313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.12.110.194:36157)
0812 11:28:46.583571 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:46.583577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:46.583599 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:28:46.583792 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:46.583803 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:46.583869 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:46.583982 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:46.585272 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.585991 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.587364 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.587765 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.587902 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.596177 (+ 8275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:46.596192 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:46.596207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:46.596241 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:46.599312 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:46.599975 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:46.599982 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:46.599985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:46.600113 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:46.600508 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:46.600512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:46.600514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:46.600870 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:46.601213 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:46.601554 (+ 341us) server_negotiation.cc:299] Negotiation successful
0812 11:28:46.601772 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260812 11:28:46.602748 10692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:46.695200 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 10483
I20260812 11:28:46.724152 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 10715
Aug 12 11:28:46 dist-test-slave-h6n0 krb5kdc[10060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:28:46.740878 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:46.725087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.12.110.195:45561)
0812 11:28:46.725306 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:46.725313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:46.725412 (+ 99us) server_negotiation.cc:407] Connection header received
0812 11:28:46.725658 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:46.725663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:46.725738 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:46.725848 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:46.727238 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.727970 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.729387 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:46.729786 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:46.729934 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:46.734911 (+ 4977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:46.734925 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:46.734942 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:46.734981 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:46.737847 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:46.738617 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:46.738624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:46.738627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:46.738766 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:46.739172 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:46.739178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:46.739182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:46.739684 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:46.739992 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:46.740290 (+ 298us) server_negotiation.cc:299] Negotiation successful
0812 11:28:46.740529 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 11:28:46.741750 10928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:46.829545 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 10715
I20260812 11:28:46.852996 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 10083
I20260812 11:28:46.854312 10198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:46.967577 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 10083
2026-08-12T11:28:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5125 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-12T11:28:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:28:47Z Disabled control of system clock
I20260812 11:28:47.066474 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:35613
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:35613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:28:47.407760 11041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:47.408037 11041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:47.408232 11041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:47.415262 11041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:28:47.415361 11041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:47.415465 11041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:47.415570 11041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:28:47.415657 11041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:28:47.424510 11041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:35613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:35613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:47.425678 11041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:47.427429 11041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:47.439193 11050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:47.439221 11051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:47.439308 11055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:47.440338 11041 server_base.cc:1034] running on GCE node
I20260812 11:28:47.441113 11041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:47.442400 11041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:47.443645 11041 hybrid_clock.cc:648] HybridClock initialized: now 1786534127443583 us; error 47 us; skew 500 ppm
I20260812 11:28:47.444000 11041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:47.446331 11041 webserver.cc:460] Webserver started at http://127.12.110.254:33117/ using document root <none> and password file <none>
I20260812 11:28:47.447074 11041 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:47.447209 11041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:47.447554 11041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:47.450392 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "6cea8f3f0b574411bf329d214e1bca82"
format_stamp: "Formatted at 2026-08-12 11:28:47 on dist-test-slave-h6n0"
server_key: "13c6bbbf4712026e2e5be52d3be900e8"
server_key_iv: "7541f71aba9d01a769da6588b5dc9df2"
server_key_version: "encryptionkey@0"
I20260812 11:28:47.451285 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "6cea8f3f0b574411bf329d214e1bca82"
format_stamp: "Formatted at 2026-08-12 11:28:47 on dist-test-slave-h6n0"
server_key: "13c6bbbf4712026e2e5be52d3be900e8"
server_key_iv: "7541f71aba9d01a769da6588b5dc9df2"
server_key_version: "encryptionkey@0"
I20260812 11:28:47.456107 11041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260812 11:28:47.459458 11061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:47.460673 11041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:47.460861 11041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "6cea8f3f0b574411bf329d214e1bca82"
format_stamp: "Formatted at 2026-08-12 11:28:47 on dist-test-slave-h6n0"
server_key: "13c6bbbf4712026e2e5be52d3be900e8"
server_key_iv: "7541f71aba9d01a769da6588b5dc9df2"
server_key_version: "encryptionkey@0"
I20260812 11:28:47.461081 11041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:47.484040 11041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:47.484937 11041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:47.485236 11041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:47.507990 11041 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:35613
I20260812 11:28:47.508044 11154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:35613 every 8 connection(s)
I20260812 11:28:47.509480 11041 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:28:47.515081 11156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:47.515755 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 11041
I20260812 11:28:47.515964 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:28:47.516434 12731 external_mini_cluster.cc:1428] Setting key 39ec91956d3828440471cf0711c32ac2
I20260812 11:28:47.531598 11156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82: Bootstrap starting.
I20260812 11:28:47.534003 11164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:47.518486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.12.110.254:35613)
0812 11:28:47.519731 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:47.519748 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:47.519801 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:28:47.522705 (+ 2904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:47.522733 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:47.523336 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:47.523772 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:47.524308 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:47.526234 (+ 1926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:47.527752 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:47.528883 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:47.529057 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:47.530581 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:47.530599 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:47.530625 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:47.531705 (+ 1080us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:47.531839 (+ 134us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:47.531874 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:47.532039 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:47.532274 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 11:28:47.532630 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":401,"threads_started":1}
I20260812 11:28:47.536289 11156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:47.537590 11156 log.cc:824] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:47.540632 11156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82: No bootstrap required, opened a new log
I20260812 11:28:47.548624 11156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea8f3f0b574411bf329d214e1bca82" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35613 } }
I20260812 11:28:47.548956 11156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:47.549031 11156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cea8f3f0b574411bf329d214e1bca82, State: Initialized, Role: FOLLOWER
I20260812 11:28:47.549520 11156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea8f3f0b574411bf329d214e1bca82" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35613 } }
I20260812 11:28:47.549724 11156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:28:47.549823 11156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:28:47.549934 11156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:47.552053 11156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea8f3f0b574411bf329d214e1bca82" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35613 } }
I20260812 11:28:47.552518 11156 leader_election.cc:302] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cea8f3f0b574411bf329d214e1bca82; no voters:
I20260812 11:28:47.553025 11156 leader_election.cc:288] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:28:47.553193 11166 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:47.554589 11166 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [term 1 LEADER]: Becoming Leader. State: Replica: 6cea8f3f0b574411bf329d214e1bca82, State: Running, Role: LEADER
I20260812 11:28:47.555198 11156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:28:47.555073 11166 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea8f3f0b574411bf329d214e1bca82" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35613 } }
I20260812 11:28:47.557853 11170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cea8f3f0b574411bf329d214e1bca82. Latest consensus state: current_term: 1 leader_uuid: "6cea8f3f0b574411bf329d214e1bca82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea8f3f0b574411bf329d214e1bca82" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35613 } } }
I20260812 11:28:47.558386 11170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:47.557879 11168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cea8f3f0b574411bf329d214e1bca82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cea8f3f0b574411bf329d214e1bca82" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35613 } } }
I20260812 11:28:47.558624 11168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82 [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:47.563160 11174 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:28:47.568087 11174 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:28:47.577423 11174 catalog_manager.cc:1347] Generated new cluster ID: 1a9955307b0d422fb5e01a6cd38e0598
I20260812 11:28:47.577533 11174 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:28:47.608192 11174 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:28:47.610029 11174 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:28:47.620570 11174 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82: Generated new TSK 0
I20260812 11:28:47.621426 11174 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:28:47.638231 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:35613
--builtin_ntp_servers=127.12.110.212:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:28:47.983672 11196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:47.983932 11196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:47.984072 11196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:47.991331 11196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:47.991426 11196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:47.991596 11196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:28:48.001575 11196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:35613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:48.002934 11196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:48.004666 11196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:48.017601 11206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:48.017561 11209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:48.017640 11211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:48.018639 11196 server_base.cc:1034] running on GCE node
I20260812 11:28:48.019253 11196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:48.020171 11196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:48.021389 11196 hybrid_clock.cc:648] HybridClock initialized: now 1786534128021325 us; error 60 us; skew 500 ppm
I20260812 11:28:48.021711 11196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:48.023969 11196 webserver.cc:460] Webserver started at http://127.12.110.193:45931/ using document root <none> and password file <none>
I20260812 11:28:48.024621 11196 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:48.024736 11196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:48.025074 11196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:48.027861 11196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd"
format_stamp: "Formatted at 2026-08-12 11:28:48 on dist-test-slave-h6n0"
server_key: "1c3c8da1f218e53d13859f342ec66ef6"
server_key_iv: "5d11ac54eeb54d2af6bbb27e4a15a352"
server_key_version: "encryptionkey@0"
I20260812 11:28:48.028638 11196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd"
format_stamp: "Formatted at 2026-08-12 11:28:48 on dist-test-slave-h6n0"
server_key: "1c3c8da1f218e53d13859f342ec66ef6"
server_key_iv: "5d11ac54eeb54d2af6bbb27e4a15a352"
server_key_version: "encryptionkey@0"
I20260812 11:28:48.033613 11196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 11:28:48.037050 11219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:48.038750 11196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:48.038939 11196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd"
format_stamp: "Formatted at 2026-08-12 11:28:48 on dist-test-slave-h6n0"
server_key: "1c3c8da1f218e53d13859f342ec66ef6"
server_key_iv: "5d11ac54eeb54d2af6bbb27e4a15a352"
server_key_version: "encryptionkey@0"
I20260812 11:28:48.039157 11196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:48.095286 11196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:48.096141 11196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:48.096463 11196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:48.097447 11196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:48.098995 11196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:48.099118 11196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:48.099249 11196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:48.099344 11196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:48.135061 11196 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:43919
I20260812 11:28:48.135154 11413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:43919 every 8 connection(s)
I20260812 11:28:48.136649 11196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:28:48.142988 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 11196
I20260812 11:28:48.143230 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:28:48.143872 12731 external_mini_cluster.cc:1428] Setting key 3616a78bd832cf1739afb51e04ec44dc
I20260812 11:28:48.146845 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:35613
--builtin_ntp_servers=127.12.110.212:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:28:48.149629 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:48.139282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:36057 (local address 127.12.110.254:35613)
0812 11:28:48.139722 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:48.139727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:48.140823 (+ 1096us) server_negotiation.cc:407] Connection header received
0812 11:28:48.142812 (+ 1989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:48.142818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:48.142911 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:48.143063 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:48.144267 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.145117 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.146700 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.147192 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.147327 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:48.148750 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:48.148763 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:48.148766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:48.148784 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:48.148825 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:48.148838 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:48.148939 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:48.149163 (+ 224us) server_negotiation.cc:299] Negotiation successful
0812 11:28:48.149260 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":245,"threads_started":1}
I20260812 11:28:48.150629 11417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:48.139733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:35613 (local address 127.12.110.193:36057)
0812 11:28:48.140648 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:48.140688 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:48.142474 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:48.143259 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:48.143270 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:48.143408 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:48.144101 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:48.144121 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.145283 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.145288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:48.146497 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:48.146509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.147423 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.147431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:48.147677 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:48.148418 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:28:48.148421 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:48.148458 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:28:48.148535 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:48.148941 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:48.148956 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:48.148964 (+ 8us) client_negotiation.cc:769] Sending connection context
0812 11:28:48.149161 (+ 197us) client_negotiation.cc:240] Negotiation successful
0812 11:28:48.149526 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":305,"threads_started":1}
I20260812 11:28:48.152212 11414 heartbeater.cc:347] Connected to a master server at 127.12.110.254:35613
I20260812 11:28:48.152506 11414 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:48.153216 11414 heartbeater.cc:511] Master 127.12.110.254:35613 requested a full tablet report, sending...
I20260812 11:28:48.155576 11088 ts_manager.cc:194] Registered new tserver with Master: 65d8ab0a8cb342148ef4dfd95f16d6fd (127.12.110.193:43919)
I20260812 11:28:48.157660 11088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.193:36057
W20260812 11:28:48.512110 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:48.512451 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:48.512614 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:48.520093 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:48.520232 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:48.520394 11418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:28:48.529505 11418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:35613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:48.530928 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:48.532696 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:48.546123 11433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:48.546144 11431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:48.546211 11430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:48.547006 11418 server_base.cc:1034] running on GCE node
I20260812 11:28:48.547585 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:48.548491 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:48.549729 11418 hybrid_clock.cc:648] HybridClock initialized: now 1786534128549694 us; error 54 us; skew 500 ppm
I20260812 11:28:48.550055 11418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:48.552750 11418 webserver.cc:460] Webserver started at http://127.12.110.194:43753/ using document root <none> and password file <none>
I20260812 11:28:48.553431 11418 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:48.553563 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:48.553872 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:48.556876 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "69e89eb138584d71b65a6501c9936ec4"
format_stamp: "Formatted at 2026-08-12 11:28:48 on dist-test-slave-h6n0"
server_key: "66c4b0253c3e5be55b505e83b896efb9"
server_key_iv: "fe16edabe580705c4b143a8cac30b6e3"
server_key_version: "encryptionkey@0"
I20260812 11:28:48.557652 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "69e89eb138584d71b65a6501c9936ec4"
format_stamp: "Formatted at 2026-08-12 11:28:48 on dist-test-slave-h6n0"
server_key: "66c4b0253c3e5be55b505e83b896efb9"
server_key_iv: "fe16edabe580705c4b143a8cac30b6e3"
server_key_version: "encryptionkey@0"
I20260812 11:28:48.562641 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:28:48.566974 11442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:48.568224 11418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260812 11:28:48.568406 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "69e89eb138584d71b65a6501c9936ec4"
format_stamp: "Formatted at 2026-08-12 11:28:48 on dist-test-slave-h6n0"
server_key: "66c4b0253c3e5be55b505e83b896efb9"
server_key_iv: "fe16edabe580705c4b143a8cac30b6e3"
server_key_version: "encryptionkey@0"
I20260812 11:28:48.568636 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:48.597659 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:48.598515 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:48.598840 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:48.599915 11418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:48.601460 11418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:48.601616 11418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:48.601761 11418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:48.601857 11418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:48.635936 11418 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:38179
I20260812 11:28:48.635974 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:38179 every 8 connection(s)
I20260812 11:28:48.637251 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:28:48.642947 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 11418
I20260812 11:28:48.643136 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:28:48.643596 12731 external_mini_cluster.cc:1428] Setting key 4cee9a0f161471cf717a74a992bcc593
I20260812 11:28:48.646837 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:35613
--builtin_ntp_servers=127.12.110.212:46607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:28:48.652292 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:48.640041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:56609 (local address 127.12.110.254:35613)
0812 11:28:48.640291 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:48.640297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:48.641777 (+ 1480us) server_negotiation.cc:407] Connection header received
0812 11:28:48.643968 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:48.643975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:48.644064 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:48.644191 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:48.645350 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.646347 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.648143 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.648917 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.649067 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:48.651191 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:48.651201 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:48.651210 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:48.651239 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:48.651288 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:48.651308 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:48.651417 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:48.651676 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 11:28:48.651799 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260812 11:28:48.652956 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:48.640632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:35613 (local address 127.12.110.194:56609)
0812 11:28:48.641606 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:48.641665 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:48.643654 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:48.644469 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:48.644483 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:48.644588 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:48.645171 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:48.645194 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.646578 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.646588 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:48.647980 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:48.647993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:48.649363 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:48.649369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:48.649564 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:48.650808 (+ 1244us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:28:48.650811 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:48.650855 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:28:48.650932 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:48.651437 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:48.651452 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:48.651458 (+ 6us) client_negotiation.cc:769] Sending connection context
0812 11:28:48.651605 (+ 147us) client_negotiation.cc:240] Negotiation successful
0812 11:28:48.651888 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":311,"threads_started":1}
I20260812 11:28:48.654452 11638 heartbeater.cc:347] Connected to a master server at 127.12.110.254:35613
I20260812 11:28:48.654801 11638 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:48.655735 11638 heartbeater.cc:511] Master 127.12.110.254:35613 requested a full tablet report, sending...
I20260812 11:28:48.657568 11088 ts_manager.cc:194] Registered new tserver with Master: 69e89eb138584d71b65a6501c9936ec4 (127.12.110.194:38179)
I20260812 11:28:48.658511 11088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.194:56609
W20260812 11:28:48.985569 11644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:48.985867 11644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:48.986030 11644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:48.993108 11644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:48.993238 11644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:48.993410 11644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:28:49.002420 11644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:35613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:49.004009 11644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:49.005762 11644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:49.018595 11655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:49.018680 11650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:49.019086 11652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:49.019639 11644 server_base.cc:1034] running on GCE node
I20260812 11:28:49.020200 11644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:49.021116 11644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:49.022344 11644 hybrid_clock.cc:648] HybridClock initialized: now 1786534129022293 us; error 38 us; skew 500 ppm
I20260812 11:28:49.022707 11644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:49.025139 11644 webserver.cc:460] Webserver started at http://127.12.110.195:36873/ using document root <none> and password file <none>
I20260812 11:28:49.025802 11644 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:49.025939 11644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:49.026260 11644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:49.028980 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "1424abad687e48aba4409ea2d768a3d1"
format_stamp: "Formatted at 2026-08-12 11:28:49 on dist-test-slave-h6n0"
server_key: "ea5e8653af42076cf320fb90b637009e"
server_key_iv: "6e9e2f0bc5cfb4c016c6ece670b1a570"
server_key_version: "encryptionkey@0"
I20260812 11:28:49.029762 11644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "1424abad687e48aba4409ea2d768a3d1"
format_stamp: "Formatted at 2026-08-12 11:28:49 on dist-test-slave-h6n0"
server_key: "ea5e8653af42076cf320fb90b637009e"
server_key_iv: "6e9e2f0bc5cfb4c016c6ece670b1a570"
server_key_version: "encryptionkey@0"
I20260812 11:28:49.034456 11644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260812 11:28:49.037806 11665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:49.039245 11644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:49.039418 11644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "1424abad687e48aba4409ea2d768a3d1"
format_stamp: "Formatted at 2026-08-12 11:28:49 on dist-test-slave-h6n0"
server_key: "ea5e8653af42076cf320fb90b637009e"
server_key_iv: "6e9e2f0bc5cfb4c016c6ece670b1a570"
server_key_version: "encryptionkey@0"
I20260812 11:28:49.039641 11644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:49.075150 11644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:49.075992 11644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:49.076309 11644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:49.077329 11644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:49.078850 11644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:49.078971 11644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:49.079105 11644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:49.079192 11644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:49.116654 11644 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:43743
I20260812 11:28:49.116719 11855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:43743 every 8 connection(s)
I20260812 11:28:49.118002 11644 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:28:49.122933 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 11644
I20260812 11:28:49.123108 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:28:49.123576 12731 external_mini_cluster.cc:1428] Setting key c074ac7985682d46d90ad1ba9c1d2ab4
I20260812 11:28:49.130556 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.120736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:52719 (local address 127.12.110.254:35613)
0812 11:28:49.120935 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.120940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.122243 (+ 1303us) server_negotiation.cc:407] Connection header received
0812 11:28:49.124187 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.124193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.124286 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.124401 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:49.125389 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.126149 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.127711 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.128130 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.128259 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.129658 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:49.129663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:49.129667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:49.129683 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:49.129735 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:49.129748 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:49.129866 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.130101 (+ 235us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.130196 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:28:49.132069 11860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.121220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:35613 (local address 127.12.110.195:52719)
0812 11:28:49.122089 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:49.122126 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:49.123936 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:49.124613 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:49.124620 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:49.124695 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:49.125232 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.125248 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.126294 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.126304 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.127566 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.127579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.128362 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.128370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.128591 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.129324 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:28:49.129326 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:49.129378 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:28:49.129453 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:49.129866 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:49.129878 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:49.129889 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 11:28:49.130084 (+ 195us) client_negotiation.cc:240] Negotiation successful
0812 11:28:49.130452 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":286,"threads_started":1}
I20260812 11:28:49.133337 11857 heartbeater.cc:347] Connected to a master server at 127.12.110.254:35613
I20260812 11:28:49.133647 11857 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:49.134384 11857 heartbeater.cc:511] Master 127.12.110.254:35613 requested a full tablet report, sending...
I20260812 11:28:49.135890 11088 ts_manager.cc:194] Registered new tserver with Master: 1424abad687e48aba4409ea2d768a3d1 (127.12.110.195:43743)
I20260812 11:28:49.136966 11088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.195:52719
I20260812 11:28:49.138602 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:28:49.149432 11416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.143401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.12.110.254:35613)
0812 11:28:49.143574 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.143578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.144040 (+ 462us) server_negotiation.cc:407] Connection header received
0812 11:28:49.144178 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.144182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.144241 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.144325 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:49.144909 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.145603 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.147119 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.147527 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.147636 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.148599 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:49.148604 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:49.148607 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:49.148622 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:49.148651 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:49.148662 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:49.148729 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.149109 (+ 380us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.149185 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:28:49.153141 11088 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:28:49.160346 11088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:28:49.160539 11414 heartbeater.cc:503] Master 127.12.110.254:35613 was elected leader, sending a full tablet report...
I20260812 11:28:49.201633 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.177070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:43743 (local address 127.0.0.1:46698)
0812 11:28:49.193025 (+ 15955us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:49.193048 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:49.193261 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:49.193840 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:49.193845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:49.193871 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:49.194231 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.194241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.196543 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.196547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.198335 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.198350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.200523 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.200530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.200775 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.200817 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:28:49.200972 (+ 155us) client_negotiation.cc:240] Negotiation successful
0812 11:28:49.201173 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15809,"mutex_wait_us":55,"thread_start_us":13295,"threads_started":1}
I20260812 11:28:49.201835 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.174738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.12.110.193:43919)
0812 11:28:49.189893 (+ 15155us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.189901 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.190845 (+ 944us) server_negotiation.cc:407] Connection header received
0812 11:28:49.191363 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.191372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.191637 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.191814 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:49.193387 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.194925 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.199227 (+ 4302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.200785 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.201000 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.201211 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.201348 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.201504 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14990,"thread_start_us":14851,"threads_started":1}
I20260812 11:28:49.201740 11885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.175070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.12.110.194:38179)
0812 11:28:49.189082 (+ 14012us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.189108 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.190991 (+ 1883us) server_negotiation.cc:407] Connection header received
0812 11:28:49.191373 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.191380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.191637 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.191819 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:49.193150 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.194899 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.198215 (+ 3316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.200140 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.200321 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.200545 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.201175 (+ 630us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.201368 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13741,"thread_start_us":14047,"threads_started":1}
I20260812 11:28:49.202369 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.177751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.12.110.195:43743)
0812 11:28:49.190447 (+ 12696us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.190455 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.193154 (+ 2699us) server_negotiation.cc:407] Connection header received
0812 11:28:49.193496 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.193502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.193666 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.193849 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:49.194422 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.196313 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.198572 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.199919 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.200118 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.200390 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.201913 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.202069 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12509,"thread_start_us":12355,"threads_started":1}
I20260812 11:28:49.201805 11888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.174714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:43919 (local address 127.0.0.1:42072)
0812 11:28:49.190627 (+ 15913us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:49.190702 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:49.191068 (+ 366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:49.192164 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:49.192172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:49.192222 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:49.193198 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.193211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.196518 (+ 3307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.196524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.198795 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.198811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.201000 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.201006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.201183 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.201212 (+ 29us) client_negotiation.cc:769] Sending connection context
0812 11:28:49.201299 (+ 87us) client_negotiation.cc:240] Negotiation successful
0812 11:28:49.201418 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15559,"mutex_wait_us":103,"spinlock_wait_cycles":6656,"thread_start_us":17934,"threads_started":1}
I20260812 11:28:49.202822 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.174914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:38179 (local address 127.0.0.1:42638)
0812 11:28:49.190882 (+ 15968us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:49.190900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:49.191080 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:49.192164 (+ 1084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:49.192172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:49.192222 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:49.192947 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.192961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.195112 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.195118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.197961 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.197979 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.200695 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.200703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.200933 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.200982 (+ 49us) client_negotiation.cc:769] Sending connection context
0812 11:28:49.202255 (+ 1273us) client_negotiation.cc:240] Negotiation successful
0812 11:28:49.202368 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15850,"mutex_wait_us":52,"spinlock_wait_cycles":8832,"thread_start_us":17474,"threads_started":1}
I20260812 11:28:49.206456 11739 tablet_service.cc:1467] Processing CreateTablet for tablet c9bc183fe8a54f428ebc5336a94ccba9 (DEFAULT_TABLE table=test-table [id=36630dbf486d4899814ed86cd49e0fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:49.206494 11524 tablet_service.cc:1467] Processing CreateTablet for tablet c9bc183fe8a54f428ebc5336a94ccba9 (DEFAULT_TABLE table=test-table [id=36630dbf486d4899814ed86cd49e0fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:49.206494 11295 tablet_service.cc:1467] Processing CreateTablet for tablet c9bc183fe8a54f428ebc5336a94ccba9 (DEFAULT_TABLE table=test-table [id=36630dbf486d4899814ed86cd49e0fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:49.207778 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bc183fe8a54f428ebc5336a94ccba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:49.207830 11295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bc183fe8a54f428ebc5336a94ccba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:49.207844 11524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9bc183fe8a54f428ebc5336a94ccba9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:49.219688 11890 tablet_bootstrap.cc:492] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Bootstrap starting.
I20260812 11:28:49.224464 11890 tablet_bootstrap.cc:654] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:49.225622 11890 log.cc:824] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:49.228437 11892 tablet_bootstrap.cc:492] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Bootstrap starting.
I20260812 11:28:49.228435 11891 tablet_bootstrap.cc:492] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Bootstrap starting.
I20260812 11:28:49.231583 11890 tablet_bootstrap.cc:492] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: No bootstrap required, opened a new log
I20260812 11:28:49.231861 11890 ts_tablet_manager.cc:1397] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260812 11:28:49.233564 11891 tablet_bootstrap.cc:654] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:49.233744 11892 tablet_bootstrap.cc:654] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:49.235400 11892 log.cc:824] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:49.235399 11891 log.cc:824] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:49.239009 11891 tablet_bootstrap.cc:492] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: No bootstrap required, opened a new log
I20260812 11:28:49.239318 11892 tablet_bootstrap.cc:492] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: No bootstrap required, opened a new log
I20260812 11:28:49.239323 11891 ts_tablet_manager.cc:1397] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260812 11:28:49.239586 11892 ts_tablet_manager.cc:1397] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260812 11:28:49.241159 11890 raft_consensus.cc:348] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.241593 11890 raft_consensus.cc:374] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:49.241703 11890 raft_consensus.cc:729] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1424abad687e48aba4409ea2d768a3d1, State: Initialized, Role: FOLLOWER
I20260812 11:28:49.242376 11890 consensus_queue.cc:260] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.243846 11857 heartbeater.cc:503] Master 127.12.110.254:35613 was elected leader, sending a full tablet report...
I20260812 11:28:49.244396 11890 ts_tablet_manager.cc:1428] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260812 11:28:49.248548 11891 raft_consensus.cc:348] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.248994 11891 raft_consensus.cc:374] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:49.249101 11891 raft_consensus.cc:729] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d8ab0a8cb342148ef4dfd95f16d6fd, State: Initialized, Role: FOLLOWER
I20260812 11:28:49.249650 11891 consensus_queue.cc:260] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.249488 11892 raft_consensus.cc:348] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.249909 11892 raft_consensus.cc:374] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:49.250027 11892 raft_consensus.cc:729] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e89eb138584d71b65a6501c9936ec4, State: Initialized, Role: FOLLOWER
I20260812 11:28:49.250619 11892 consensus_queue.cc:260] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.252242 11891 ts_tablet_manager.cc:1428] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260812 11:28:49.252884 11638 heartbeater.cc:503] Master 127.12.110.254:35613 was elected leader, sending a full tablet report...
I20260812 11:28:49.253496 11892 ts_tablet_manager.cc:1428] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
W20260812 11:28:49.369877 11859 tablet.cc:2369] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:28:49.373298 11900 raft_consensus.cc:482] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:49.373641 11900 raft_consensus.cc:504] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.375416 11900 leader_election.cc:288] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69e89eb138584d71b65a6501c9936ec4 (127.12.110.194:38179), 1424abad687e48aba4409ea2d768a3d1 (127.12.110.195:43743)
I20260812 11:28:49.383272 11885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.375931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:48211 (local address 127.12.110.194:38179)
0812 11:28:49.376152 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.376161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.376661 (+ 500us) server_negotiation.cc:407] Connection header received
0812 11:28:49.376886 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.376890 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.376973 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.377079 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:49.377651 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.378347 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.381050 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.382293 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.382445 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.382516 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.382871 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.383017 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260812 11:28:49.383320 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.375708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:38179 (local address 127.12.110.193:48211)
0812 11:28:49.376551 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:49.376568 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:49.376740 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:49.377098 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:49.377103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:49.377128 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:49.377504 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.377516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.378490 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.378495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.380795 (+ 2300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.380807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.382496 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.382501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.382698 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.382724 (+ 26us) client_negotiation.cc:769] Sending connection context
0812 11:28:49.382808 (+ 84us) client_negotiation.cc:240] Negotiation successful
0812 11:28:49.382897 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":94,"spinlock_wait_cycles":31616,"thread_start_us":483,"threads_started":1}
I20260812 11:28:49.383452 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.376045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:53019 (local address 127.12.110.195:43743)
0812 11:28:49.376268 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.376275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.376803 (+ 528us) server_negotiation.cc:407] Connection header received
0812 11:28:49.377001 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.377005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.377063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.377161 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:49.377784 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.378484 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.381053 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.382312 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.382436 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.382505 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.383050 (+ 545us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.383200 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:28:49.383714 11903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.375875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:43743 (local address 127.12.110.193:53019)
0812 11:28:49.376691 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:49.376705 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:49.376852 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:49.377177 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:49.377186 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:49.377210 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:49.377630 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.377639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.378728 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.378733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.380784 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:49.380796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.382691 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.382696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:49.382873 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.382904 (+ 31us) client_negotiation.cc:769] Sending connection context
0812 11:28:49.383005 (+ 101us) client_negotiation.cc:240] Negotiation successful
0812 11:28:49.383122 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":4,"thread_start_us":461,"threads_started":1}
I20260812 11:28:49.384549 11559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bc183fe8a54f428ebc5336a94ccba9" candidate_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e89eb138584d71b65a6501c9936ec4" is_pre_election: true
I20260812 11:28:49.384635 11776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bc183fe8a54f428ebc5336a94ccba9" candidate_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1424abad687e48aba4409ea2d768a3d1" is_pre_election: true
I20260812 11:28:49.385152 11776 raft_consensus.cc:2454] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d8ab0a8cb342148ef4dfd95f16d6fd in term 0.
I20260812 11:28:49.385123 11559 raft_consensus.cc:2454] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d8ab0a8cb342148ef4dfd95f16d6fd in term 0.
I20260812 11:28:49.385787 11226 leader_election.cc:302] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1424abad687e48aba4409ea2d768a3d1, 65d8ab0a8cb342148ef4dfd95f16d6fd, 69e89eb138584d71b65a6501c9936ec4; no voters:
I20260812 11:28:49.386207 11900 raft_consensus.cc:2781] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:28:49.386328 11900 raft_consensus.cc:482] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:28:49.386440 11900 raft_consensus.cc:3037] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:49.388630 11900 raft_consensus.cc:504] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
W20260812 11:28:49.389194 11415 tablet.cc:2369] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:28:49.389514 11639 tablet.cc:2369] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:28:49.389783 11776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bc183fe8a54f428ebc5336a94ccba9" candidate_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1424abad687e48aba4409ea2d768a3d1"
I20260812 11:28:49.389871 11559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9bc183fe8a54f428ebc5336a94ccba9" candidate_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e89eb138584d71b65a6501c9936ec4"
I20260812 11:28:49.389992 11776 raft_consensus.cc:3037] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:49.390097 11900 leader_election.cc:288] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [CANDIDATE]: Term 1 election: Requested vote from peers 69e89eb138584d71b65a6501c9936ec4 (127.12.110.194:38179), 1424abad687e48aba4409ea2d768a3d1 (127.12.110.195:43743)
I20260812 11:28:49.390096 11559 raft_consensus.cc:3037] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:49.392230 11776 raft_consensus.cc:2454] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d8ab0a8cb342148ef4dfd95f16d6fd in term 1.
I20260812 11:28:49.392513 11559 raft_consensus.cc:2454] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d8ab0a8cb342148ef4dfd95f16d6fd in term 1.
I20260812 11:28:49.392652 11221 leader_election.cc:302] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1424abad687e48aba4409ea2d768a3d1, 65d8ab0a8cb342148ef4dfd95f16d6fd; no voters:
I20260812 11:28:49.392978 11900 raft_consensus.cc:2781] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:49.393440 11900 raft_consensus.cc:686] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 1 LEADER]: Becoming Leader. State: Replica: 65d8ab0a8cb342148ef4dfd95f16d6fd, State: Running, Role: LEADER
I20260812 11:28:49.393884 11900 consensus_queue.cc:237] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } }
I20260812 11:28:49.398375 11086 catalog_manager.cc:5495] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 65d8ab0a8cb342148ef4dfd95f16d6fd (127.12.110.193). New cstate: current_term: 1 leader_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d8ab0a8cb342148ef4dfd95f16d6fd" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 43919 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 } health_report { overall_health: UNKNOWN } } }
I20260812 11:28:49.439178 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.433875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.12.110.193:43919)
0812 11:28:49.434116 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.434122 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.434145 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:28:49.434260 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.434264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.434362 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.434525 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:28:49.434933 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.435645 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.437043 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.437452 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.437597 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.438063 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:28:49.438591 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:28:49.438748 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.438836 (+ 88us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.438915 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260812 11:28:49.447834 11776 raft_consensus.cc:1261] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 1 FOLLOWER]: Refusing update from remote peer 65d8ab0a8cb342148ef4dfd95f16d6fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:49.447896 11559 raft_consensus.cc:1261] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 1 FOLLOWER]: Refusing update from remote peer 65d8ab0a8cb342148ef4dfd95f16d6fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:49.448939 11908 consensus_queue.cc:1035] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "1424abad687e48aba4409ea2d768a3d1" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 43743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:49.449419 11900 consensus_queue.cc:1035] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e89eb138584d71b65a6501c9936ec4" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 38179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:49.479592 11086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:49.479866 11086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:49.483398 11086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6cea8f3f0b574411bf329d214e1bca82: Sending DeleteTablet for 3 replicas of tablet c9bc183fe8a54f428ebc5336a94ccba9
I20260812 11:28:49.484432 11295 tablet_service.cc:1514] Processing DeleteTablet for tablet c9bc183fe8a54f428ebc5336a94ccba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:49 UTC) from {username='slave'} at 127.0.0.1:42072
I20260812 11:28:49.484578 11524 tablet_service.cc:1514] Processing DeleteTablet for tablet c9bc183fe8a54f428ebc5336a94ccba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:49 UTC) from {username='slave'} at 127.0.0.1:42638
I20260812 11:28:49.484843 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet c9bc183fe8a54f428ebc5336a94ccba9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:49 UTC) from {username='slave'} at 127.0.0.1:46698
I20260812 11:28:49.485482 11929 tablet_replica.cc:330] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: stopping tablet replica
I20260812 11:28:49.485661 11930 tablet_replica.cc:330] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: stopping tablet replica
I20260812 11:28:49.486001 11929 raft_consensus.cc:2229] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:28:49.486054 11930 raft_consensus.cc:2229] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:49.486600 11930 raft_consensus.cc:2258] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:49.486744 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 11196
I20260812 11:28:49.486909 11929 raft_consensus.cc:2258] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:49.489681 11930 ts_tablet_manager.cc:1905] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:49.500483 11929 ts_tablet_manager.cc:1905] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:49.501602 11931 tablet_replica.cc:330] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: stopping tablet replica
I20260812 11:28:49.502063 11931 raft_consensus.cc:2229] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:49.502636 11931 raft_consensus.cc:2258] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:49.504916 11931 ts_tablet_manager.cc:1905] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:49.508666 11929 ts_tablet_manager.cc:1918] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:49.508854 11929 log.cc:1196] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/c9bc183fe8a54f428ebc5336a94ccba9
I20260812 11:28:49.509459 11929 ts_tablet_manager.cc:1939] T c9bc183fe8a54f428ebc5336a94ccba9 P 65d8ab0a8cb342148ef4dfd95f16d6fd: Deleting consensus metadata
I20260812 11:28:49.511411 11063 catalog_manager.cc:4841] TS 65d8ab0a8cb342148ef4dfd95f16d6fd (127.12.110.193:43919): tablet c9bc183fe8a54f428ebc5336a94ccba9 (table test-table [id=36630dbf486d4899814ed86cd49e0fb6]) successfully deleted
I20260812 11:28:49.512089 11930 ts_tablet_manager.cc:1918] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:49.512279 11930 log.cc:1196] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/c9bc183fe8a54f428ebc5336a94ccba9
I20260812 11:28:49.512382 11931 ts_tablet_manager.cc:1918] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:49.512550 11931 log.cc:1196] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/c9bc183fe8a54f428ebc5336a94ccba9
I20260812 11:28:49.512889 11930 ts_tablet_manager.cc:1939] T c9bc183fe8a54f428ebc5336a94ccba9 P 69e89eb138584d71b65a6501c9936ec4: Deleting consensus metadata
I20260812 11:28:49.513520 11931 ts_tablet_manager.cc:1939] T c9bc183fe8a54f428ebc5336a94ccba9 P 1424abad687e48aba4409ea2d768a3d1: Deleting consensus metadata
I20260812 11:28:49.514914 11067 catalog_manager.cc:4841] TS 69e89eb138584d71b65a6501c9936ec4 (127.12.110.194:38179): tablet c9bc183fe8a54f428ebc5336a94ccba9 (table test-table [id=36630dbf486d4899814ed86cd49e0fb6]) successfully deleted
I20260812 11:28:49.515148 11062 catalog_manager.cc:4841] TS 1424abad687e48aba4409ea2d768a3d1 (127.12.110.195:43743): tablet c9bc183fe8a54f428ebc5336a94ccba9 (table test-table [id=36630dbf486d4899814ed86cd49e0fb6]) successfully deleted
I20260812 11:28:49.516110 11880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.488305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42086 (local address 127.12.110.193:43919)
0812 11:28:49.488532 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.488538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.498425 (+ 9887us) server_negotiation.cc:407] Connection header received
0812 11:28:49.499377 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.499384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.499493 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.499610 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:49.500275 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.501593 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.504129 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.504849 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.504967 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.506605 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:49.506620 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:49.506640 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:49.507501 (+ 861us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:49.507570 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:49.507595 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:49.507672 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.515604 (+ 7932us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.515714 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260812 11:28:49.516871 11406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:49.620126 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 11196
I20260812 11:28:49.645826 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 11418
I20260812 11:28:49.654173 11885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.646799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.12.110.194:38179)
0812 11:28:49.646972 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.646977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.646995 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:28:49.647147 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.647150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.647225 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.647320 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:49.647730 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.648566 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.650478 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.651202 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.651308 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.652573 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:49.652581 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:49.652594 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:49.653156 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:49.653200 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:49.653222 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:49.653316 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.653496 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.653623 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260812 11:28:49.655059 11625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:49.748724 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 11418
I20260812 11:28:49.777411 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 11644
I20260812 11:28:49.784502 11889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:49.778199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.12.110.195:43743)
0812 11:28:49.778390 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:49.778398 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:49.778459 (+ 61us) server_negotiation.cc:407] Connection header received
0812 11:28:49.778706 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:49.778712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:49.778809 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:49.778917 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:49.779321 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.780045 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.781348 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:49.781806 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:49.781930 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:49.783113 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:49.783129 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:49.783144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:49.783711 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:49.783758 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:49.783777 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:49.783907 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:49.784044 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 11:28:49.784153 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:28:49.785156 11851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:49.874461 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 11644
I20260812 11:28:49.899950 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 11041
I20260812 11:28:49.901026 11147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:50.011607 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 11041
2026-08-12T11:28:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3045 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:28:50 dist-test-slave-h6n0 krb5kdc[11941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:28:50 dist-test-slave-h6n0 krb5kdc[11941](info): set up 2 sockets
Aug 12 11:28:50 dist-test-slave-h6n0 krb5kdc[11941](info): commencing operation
krb5kdc: starting...
W20260812 11:28:51.411799 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:28:51 dist-test-slave-h6n0 krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:28:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:28:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:28:51.758765 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42705
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:42705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:52.108525 11964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:52.108855 11964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:52.109022 11964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:52.116103 11964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:28:52.116251 11964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:52.116357 11964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:52.116469 11964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:28:52.116547 11964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:28:52.126389 11964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:42705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:42705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:52.127640 11964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:52.128939 11964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:52.140861 11970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:52.140959 11974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:52.140910 11976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:52.142007 11964 server_base.cc:1034] running on GCE node
I20260812 11:28:52.143257 11964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:52.144600 11964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:52.145821 11964 hybrid_clock.cc:648] HybridClock initialized: now 1786534132145766 us; error 44 us; skew 500 ppm
I20260812 11:28:52.146128 11964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:52 dist-test-slave-h6n0 krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:52.152578 11964 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:28:52.154134 11964 webserver.cc:460] Webserver started at http://127.12.110.254:43255/ using document root <none> and password file <none>
I20260812 11:28:52.154874 11964 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:52.155028 11964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:52.155341 11964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:52.158279 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "b3d8645ed4074056bd356970d38c1bd7"
format_stamp: "Formatted at 2026-08-12 11:28:52 on dist-test-slave-h6n0"
server_key: "0ac606a77aa0c81aa14d465401ae8152"
server_key_iv: "b398251ddc67f9721559ac7d06e78135"
server_key_version: "encryptionkey@0"
I20260812 11:28:52.159098 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "b3d8645ed4074056bd356970d38c1bd7"
format_stamp: "Formatted at 2026-08-12 11:28:52 on dist-test-slave-h6n0"
server_key: "0ac606a77aa0c81aa14d465401ae8152"
server_key_iv: "b398251ddc67f9721559ac7d06e78135"
server_key_version: "encryptionkey@0"
I20260812 11:28:52.164067 11964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:28:52.167420 11984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:52.168624 11964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:28:52.168812 11964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "b3d8645ed4074056bd356970d38c1bd7"
format_stamp: "Formatted at 2026-08-12 11:28:52 on dist-test-slave-h6n0"
server_key: "0ac606a77aa0c81aa14d465401ae8152"
server_key_iv: "b398251ddc67f9721559ac7d06e78135"
server_key_version: "encryptionkey@0"
I20260812 11:28:52.169014 11964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:52.186652 11964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:52.191097 11964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:52.191399 11964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:52.211606 11964 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:42705
I20260812 11:28:52.211629 12073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:42705 every 8 connection(s)
I20260812 11:28:52.213012 11964 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:28:52.217430 12075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:52.219480 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 11964
I20260812 11:28:52.219674 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:28:52.220228 12731 external_mini_cluster.cc:1428] Setting key 20ec2c8d508ae2308b676c7e2b84ab78
I20260812 11:28:52.230984 12075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7: Bootstrap starting.
I20260812 11:28:52.234010 12075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7: Neither blocks nor log segments found. Creating new log.
Aug 12 11:28:52 dist-test-slave-h6n0 krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:52.235284 12075 log.cc:824] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:52.239485 12075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7: No bootstrap required, opened a new log
I20260812 11:28:52.245378 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:52.222293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.12.110.254:42705)
0812 11:28:52.223169 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:52.223180 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:52.223226 (+ 46us) server_negotiation.cc:407] Connection header received
0812 11:28:52.224842 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:52.224863 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:52.225238 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:52.225695 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:52.227521 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:52.228850 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:52.230228 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:52.230966 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:52.231161 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:52.236051 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:52.236062 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:52.236075 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:52.236110 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:52.240738 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:52.241467 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:52.241476 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:52.241485 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:52.241702 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:52.242085 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:52.242090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:52.242093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:52.242867 (+ 774us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:52.243188 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:52.243617 (+ 429us) server_negotiation.cc:299] Negotiation successful
0812 11:28:52.244060 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":433,"threads_started":1}
I20260812 11:28:52.249660 12075 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d8645ed4074056bd356970d38c1bd7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42705 } }
I20260812 11:28:52.249990 12075 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:52.250065 12075 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3d8645ed4074056bd356970d38c1bd7, State: Initialized, Role: FOLLOWER
I20260812 11:28:52.250633 12075 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d8645ed4074056bd356970d38c1bd7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42705 } }
I20260812 11:28:52.250852 12075 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:28:52.250955 12075 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:28:52.251068 12075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:52.253140 12075 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d8645ed4074056bd356970d38c1bd7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42705 } }
I20260812 11:28:52.253589 12075 leader_election.cc:302] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3d8645ed4074056bd356970d38c1bd7; no voters:
I20260812 11:28:52.254246 12075 leader_election.cc:288] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:28:52.254380 12081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:52.255273 12081 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [term 1 LEADER]: Becoming Leader. State: Replica: b3d8645ed4074056bd356970d38c1bd7, State: Running, Role: LEADER
I20260812 11:28:52.255843 12081 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d8645ed4074056bd356970d38c1bd7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42705 } }
I20260812 11:28:52.256414 12075 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:28:52.258736 12085 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3d8645ed4074056bd356970d38c1bd7. Latest consensus state: current_term: 1 leader_uuid: "b3d8645ed4074056bd356970d38c1bd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d8645ed4074056bd356970d38c1bd7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42705 } } }
I20260812 11:28:52.259260 12085 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:52.259017 12084 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3d8645ed4074056bd356970d38c1bd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3d8645ed4074056bd356970d38c1bd7" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 42705 } } }
I20260812 11:28:52.259406 12084 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7 [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:52.260263 12097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:28:52.266428 12097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:28:52.280645 12097 catalog_manager.cc:1347] Generated new cluster ID: 280443d403e340cbb63f9428c5cc7e27
I20260812 11:28:52.280761 12097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:28:52.314725 12097 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:28:52.316325 12097 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:28:52.326502 12097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7: Generated new TSK 0
I20260812 11:28:52.327486 12097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:28:52.411152 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42705
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:52.758513 12117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:52.758889 12117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:52.759065 12117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:52.767539 12117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:52.767714 12117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:52.767907 12117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:28:52.780375 12117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:52.781769 12117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:52.783190 12117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:52.796343 12126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:52.796294 12127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:52.796387 12129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:52.797163 12117 server_base.cc:1034] running on GCE node
I20260812 11:28:52.797758 12117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:52.798748 12117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:52.800009 12117 hybrid_clock.cc:648] HybridClock initialized: now 1786534132799956 us; error 41 us; skew 500 ppm
I20260812 11:28:52.800314 12117 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:52 dist-test-slave-h6n0 krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:52.803565 12117 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:28:52.805052 12117 webserver.cc:460] Webserver started at http://127.12.110.193:38511/ using document root <none> and password file <none>
I20260812 11:28:52.805704 12117 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:52.805856 12117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:52.806169 12117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:52.809021 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "2077385f81e041d2a490ab050e8f61b7"
format_stamp: "Formatted at 2026-08-12 11:28:52 on dist-test-slave-h6n0"
server_key: "1c9150abadc04c1717a41237f42a99d2"
server_key_iv: "b4d257db1295da27d880c9023eb593e2"
server_key_version: "encryptionkey@0"
I20260812 11:28:52.809770 12117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "2077385f81e041d2a490ab050e8f61b7"
format_stamp: "Formatted at 2026-08-12 11:28:52 on dist-test-slave-h6n0"
server_key: "1c9150abadc04c1717a41237f42a99d2"
server_key_iv: "b4d257db1295da27d880c9023eb593e2"
server_key_version: "encryptionkey@0"
I20260812 11:28:52.814634 12117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260812 11:28:52.817827 12136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:52.819087 12117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:52.819264 12117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "2077385f81e041d2a490ab050e8f61b7"
format_stamp: "Formatted at 2026-08-12 11:28:52 on dist-test-slave-h6n0"
server_key: "1c9150abadc04c1717a41237f42a99d2"
server_key_iv: "b4d257db1295da27d880c9023eb593e2"
server_key_version: "encryptionkey@0"
I20260812 11:28:52.819470 12117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:52.832823 12117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:52.837181 12117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:52.837482 12117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:52.838429 12117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:52.839951 12117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:52.840062 12117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:52.840214 12117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:52.840296 12117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:52.875221 12117 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:33293
I20260812 11:28:52.875324 12325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:33293 every 8 connection(s)
I20260812 11:28:52.876525 12117 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:28:52.881906 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 12117
I20260812 11:28:52.882081 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:28:52.882573 12731 external_mini_cluster.cc:1428] Setting key 36bb7a8187ea663d3d8e381dde00b3f8
Aug 12 11:28:52 dist-test-slave-h6n0 krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534132, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:52.898181 12329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:52.879159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:35167 (local address 127.12.110.254:42705)
0812 11:28:52.879609 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:52.879614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:52.880575 (+ 961us) server_negotiation.cc:407] Connection header received
0812 11:28:52.882512 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:52.882518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:52.882632 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:52.882768 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:52.884605 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:52.885315 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:52.887825 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:52.888251 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:52.888349 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:52.892534 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:52.892539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:52.892542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:52.892572 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:52.895176 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:52.895887 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:52.895894 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:52.895897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:52.896023 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:52.896466 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:52.896471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:52.896473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:52.896734 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:52.896893 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:52.897587 (+ 694us) server_negotiation.cc:299] Negotiation successful
0812 11:28:52.897810 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260812 11:28:52.899139 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:52.879674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42705 (local address 127.12.110.193:35167)
0812 11:28:52.880394 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:52.880434 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:52.882256 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:52.883036 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:52.883047 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:52.883754 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:52.884406 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:52.884426 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:52.885487 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:52.885494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:52.887663 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:52.887680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:52.888484 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:52.888494 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:52.888804 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:52.889567 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:52.889600 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:52.892317 (+ 2717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:52.895324 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:52.895332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:52.895346 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:52.895763 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:52.896137 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:52.896142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:52.896145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:52.896339 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:52.896867 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:52.896875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:52.897209 (+ 334us) client_negotiation.cc:769] Sending connection context
0812 11:28:52.897471 (+ 262us) client_negotiation.cc:240] Negotiation successful
0812 11:28:52.897972 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":299,"threads_started":1}
I20260812 11:28:52.900797 12327 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42705
I20260812 11:28:52.901126 12327 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:52.901952 12327 heartbeater.cc:511] Master 127.12.110.254:42705 requested a full tablet report, sending...
I20260812 11:28:52.904230 12013 ts_manager.cc:194] Registered new tserver with Master: 2077385f81e041d2a490ab050e8f61b7 (127.12.110.193:33293)
I20260812 11:28:52.905604 12013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:35167
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:28:52.954160 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42705
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:53.292335 12338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:53.292953 12338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:53.293210 12338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:53.302601 12338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:53.302801 12338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:53.303020 12338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:28:53.314774 12338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:53.316185 12338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:53.317505 12338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:53.333534 12351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:53.333743 12357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:53.333673 12350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:53.339879 12338 server_base.cc:1034] running on GCE node
I20260812 11:28:53.340572 12338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:53.341486 12338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:53.342657 12338 hybrid_clock.cc:648] HybridClock initialized: now 1786534133342607 us; error 33 us; skew 500 ppm
I20260812 11:28:53.342999 12338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:53 dist-test-slave-h6n0 krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534133, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:53.346341 12338 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:28:53.347955 12338 webserver.cc:460] Webserver started at http://127.12.110.194:36785/ using document root <none> and password file <none>
I20260812 11:28:53.348704 12338 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:53.348861 12338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:53.349201 12338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:53.352205 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "8d14f2f5a8814d688f7414495f86138d"
format_stamp: "Formatted at 2026-08-12 11:28:53 on dist-test-slave-h6n0"
server_key: "97f586fcc7d298caf9c4c8e13ee2b1cb"
server_key_iv: "bfb214f01e073b930d5691cfe98fa8ce"
server_key_version: "encryptionkey@0"
I20260812 11:28:53.353060 12338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "8d14f2f5a8814d688f7414495f86138d"
format_stamp: "Formatted at 2026-08-12 11:28:53 on dist-test-slave-h6n0"
server_key: "97f586fcc7d298caf9c4c8e13ee2b1cb"
server_key_iv: "bfb214f01e073b930d5691cfe98fa8ce"
server_key_version: "encryptionkey@0"
I20260812 11:28:53.358373 12338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:28:53.362221 12369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:53.363828 12338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:28:53.364017 12338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "8d14f2f5a8814d688f7414495f86138d"
format_stamp: "Formatted at 2026-08-12 11:28:53 on dist-test-slave-h6n0"
server_key: "97f586fcc7d298caf9c4c8e13ee2b1cb"
server_key_iv: "bfb214f01e073b930d5691cfe98fa8ce"
server_key_version: "encryptionkey@0"
I20260812 11:28:53.364204 12338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:53.392717 12338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:53.397251 12338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:53.397589 12338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:53.398695 12338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:53.400202 12338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:53.400280 12338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:53.400453 12338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:53.400506 12338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:53.434221 12338 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:39247
I20260812 11:28:53.434270 12552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:39247 every 8 connection(s)
I20260812 11:28:53.435667 12338 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:28:53.436018 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 12338
I20260812 11:28:53.436225 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:28:53.436820 12731 external_mini_cluster.cc:1428] Setting key bddfacd6edf8b2e0d3eee2cb14c89be1
Aug 12 11:28:53 dist-test-slave-h6n0 krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534133, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:53.458101 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:53.438430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:37695 (local address 127.12.110.254:42705)
0812 11:28:53.438908 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:53.438914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:53.440170 (+ 1256us) server_negotiation.cc:407] Connection header received
0812 11:28:53.443052 (+ 2882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:53.443056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:53.443118 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:53.443211 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:53.445255 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:53.445981 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:53.447609 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:53.448037 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:53.448188 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:53.452288 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:53.452294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:53.452297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:53.452328 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:53.454988 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:53.455696 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:53.455701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:53.455703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:53.455825 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:53.456292 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:53.456298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:53.456304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:53.456638 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:53.456814 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:53.457492 (+ 678us) server_negotiation.cc:299] Negotiation successful
0812 11:28:53.457727 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":235,"threads_started":1}
I20260812 11:28:53.459106 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:53.439111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42705 (local address 127.12.110.194:37695)
0812 11:28:53.439976 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:53.440042 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:53.442765 (+ 2723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:53.443463 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:53.443476 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:53.444269 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:53.445061 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:53.445090 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:53.446153 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:53.446157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:53.447449 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:53.447461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:53.448283 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:53.448291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:53.448509 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:53.449233 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:53.449256 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:53.452061 (+ 2805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:53.455128 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:53.455137 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:53.455152 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:53.455560 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:53.455949 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:53.455954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:53.455958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:53.456151 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:53.456772 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:53.456779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:53.457124 (+ 345us) client_negotiation.cc:769] Sending connection context
0812 11:28:53.457376 (+ 252us) client_negotiation.cc:240] Negotiation successful
0812 11:28:53.457935 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":310,"threads_started":1}
I20260812 11:28:53.460577 12553 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42705
I20260812 11:28:53.460872 12553 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:53.461612 12553 heartbeater.cc:511] Master 127.12.110.254:42705 requested a full tablet report, sending...
I20260812 11:28:53.463172 12013 ts_manager.cc:194] Registered new tserver with Master: 8d14f2f5a8814d688f7414495f86138d (127.12.110.194:39247)
I20260812 11:28:53.464444 12013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:37695
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:28:53.509857 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42705
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:53.854048 12564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:53.854357 12564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:53.854458 12564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:53.862042 12564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:53.862150 12564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:53.862378 12564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:28:53.872543 12564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:42705
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:53.873842 12564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:53.875273 12564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:53.889979 12573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:53.890025 12575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:53.890074 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:53.891319 12564 server_base.cc:1034] running on GCE node
I20260812 11:28:53.891980 12564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:53.893113 12564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:53.894394 12564 hybrid_clock.cc:648] HybridClock initialized: now 1786534133894327 us; error 61 us; skew 500 ppm
I20260812 11:28:53.894778 12564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:53 dist-test-slave-h6n0 krb5kdc[11941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534133, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:53.898110 12564 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:28:53.899859 12564 webserver.cc:460] Webserver started at http://127.12.110.195:41079/ using document root <none> and password file <none>
I20260812 11:28:53.900732 12564 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:53.900947 12564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:53.901345 12564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:53.904913 12564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "1c60ce64583041ed9fa167b0ba12c8b4"
format_stamp: "Formatted at 2026-08-12 11:28:53 on dist-test-slave-h6n0"
server_key: "25b2729d3552e2b8ddaff9a28905cdcd"
server_key_iv: "94f331baaacf428c081eb915fb493640"
server_key_version: "encryptionkey@0"
I20260812 11:28:53.905740 12564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "1c60ce64583041ed9fa167b0ba12c8b4"
format_stamp: "Formatted at 2026-08-12 11:28:53 on dist-test-slave-h6n0"
server_key: "25b2729d3552e2b8ddaff9a28905cdcd"
server_key_iv: "94f331baaacf428c081eb915fb493640"
server_key_version: "encryptionkey@0"
I20260812 11:28:53.909113 12327 heartbeater.cc:503] Master 127.12.110.254:42705 was elected leader, sending a full tablet report...
I20260812 11:28:53.911194 12564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:28:53.914693 12590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:53.916025 12564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:28:53.916201 12564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "1c60ce64583041ed9fa167b0ba12c8b4"
format_stamp: "Formatted at 2026-08-12 11:28:53 on dist-test-slave-h6n0"
server_key: "25b2729d3552e2b8ddaff9a28905cdcd"
server_key_iv: "94f331baaacf428c081eb915fb493640"
server_key_version: "encryptionkey@0"
I20260812 11:28:53.916383 12564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:53.940443 12564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:53.944952 12564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:53.945240 12564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:53.946192 12564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:53.947592 12564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:53.947666 12564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:53.947835 12564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:53.947888 12564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:53.985349 12564 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:40707
I20260812 11:28:53.985386 12778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:40707 every 8 connection(s)
I20260812 11:28:53.987215 12564 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:28:53.992143 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 12564
I20260812 11:28:53.992380 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:28:53.993072 12731 external_mini_cluster.cc:1428] Setting key 0f9858b71f78c892f785d388a32fe7e7
Aug 12 11:28:54 dist-test-slave-h6n0 krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534133, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:28:54.009697 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:53.990767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:42161 (local address 127.12.110.254:42705)
0812 11:28:53.991291 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:53.991301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:53.992734 (+ 1433us) server_negotiation.cc:407] Connection header received
0812 11:28:53.995025 (+ 2291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:53.995033 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:53.995158 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:53.995363 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:53.997329 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:53.998138 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:53.999751 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.000159 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.000254 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.003698 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:54.003703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:54.003706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:54.003736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:54.006409 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.007119 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.007125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.007129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.007281 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.007768 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.007773 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.007776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.008148 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:54.008309 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.009188 (+ 879us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.009408 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":243,"threads_started":1}
I20260812 11:28:54.010449 12785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:53.991347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:42705 (local address 127.12.110.195:42161)
0812 11:28:53.992208 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:53.992290 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:53.994524 (+ 2234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:53.995658 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:53.995669 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:53.996542 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:53.997135 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:53.997154 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:53.998288 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:53.998292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:53.999606 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:53.999619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.000381 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.000387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.000583 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.001244 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:28:54.001266 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:54.003494 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:54.006569 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:54.006580 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:54.006599 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:54.006998 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:54.007435 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:28:54.007440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:28:54.007443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:28:54.007658 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:28:54.008320 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:54.008328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:54.008765 (+ 437us) client_negotiation.cc:769] Sending connection context
0812 11:28:54.009051 (+ 286us) client_negotiation.cc:240] Negotiation successful
0812 11:28:54.009406 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":385,"threads_started":1}
I20260812 11:28:54.011873 12781 heartbeater.cc:347] Connected to a master server at 127.12.110.254:42705
I20260812 11:28:54.012135 12781 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:54.012775 12781 heartbeater.cc:511] Master 127.12.110.254:42705 requested a full tablet report, sending...
I20260812 11:28:54.014112 12013 ts_manager.cc:194] Registered new tserver with Master: 1c60ce64583041ed9fa167b0ba12c8b4 (127.12.110.195:40707)
I20260812 11:28:54.014994 12013 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:42161
I20260812 11:28:54.021801 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:28:54.041607 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.027340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60948 (local address 127.12.110.254:42705)
0812 11:28:54.027598 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.027607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.028141 (+ 534us) server_negotiation.cc:407] Connection header received
0812 11:28:54.028382 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.028386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.028459 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.028580 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:54.030391 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.031211 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.032535 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.032978 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.033145 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.036025 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:54.036030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:54.036034 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:54.036069 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:54.038783 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.039563 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.039570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.039579 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.039695 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.040163 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.040174 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.040177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.040463 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:54.040750 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.041127 (+ 377us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.041336 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:28:54.046221 12013 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:28:54.053606 12013 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:28:54.088339 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.066076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:39247 (local address 127.0.0.1:49910)
0812 11:28:54.077898 (+ 11822us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:54.077920 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:54.078069 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:54.079663 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:54.079674 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:54.079719 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:54.080288 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.080311 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.082491 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.082498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.085788 (+ 3290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.085801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.087196 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.087203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.087437 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.087479 (+ 42us) client_negotiation.cc:769] Sending connection context
0812 11:28:54.087636 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 11:28:54.087839 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11667,"mutex_wait_us":126,"thread_start_us":11523,"threads_started":1}
I20260812 11:28:54.088454 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.066199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47078 (local address 127.12.110.195:40707)
0812 11:28:54.078611 (+ 12412us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.078619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.078652 (+ 33us) server_negotiation.cc:407] Connection header received
0812 11:28:54.080228 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.080240 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.080511 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.081556 (+ 1045us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:54.081844 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.083411 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.086084 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.087493 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.087686 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.087791 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.088050 (+ 259us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.088177 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12114,"thread_start_us":11957,"threads_started":1}
I20260812 11:28:54.088919 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.066367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49910 (local address 127.12.110.194:39247)
0812 11:28:54.076441 (+ 10074us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.076453 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.078058 (+ 1605us) server_negotiation.cc:407] Connection header received
0812 11:28:54.078316 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.078329 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.078696 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.079096 (+ 400us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:54.080585 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.082355 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.085965 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.086988 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.087174 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.087362 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.088445 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.088632 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9895,"thread_start_us":9788,"threads_started":1}
I20260812 11:28:54.089033 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.066007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:40707 (local address 127.0.0.1:47078)
0812 11:28:54.076859 (+ 10852us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:54.076914 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:54.077075 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:54.080773 (+ 3698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:54.080780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:54.080814 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:54.081331 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.081346 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.083579 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.083584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.085873 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.085896 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.087706 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.087711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.087886 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.087914 (+ 28us) client_negotiation.cc:769] Sending connection context
0812 11:28:54.088018 (+ 104us) client_negotiation.cc:240] Negotiation successful
0812 11:28:54.088148 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10623,"mutex_wait_us":93,"thread_start_us":10449,"threads_started":1}
I20260812 11:28:54.091732 12815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.078603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:33293 (local address 127.0.0.1:37924)
0812 11:28:54.082146 (+ 3543us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:54.082163 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:54.082280 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:54.082835 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:54.082840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:54.082870 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:54.083265 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.083279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.085201 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.085207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.087492 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.087506 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.090900 (+ 3394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.090907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.091072 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.091112 (+ 40us) client_negotiation.cc:769] Sending connection context
0812 11:28:54.091233 (+ 121us) client_negotiation.cc:240] Negotiation successful
0812 11:28:54.091368 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3438,"mutex_wait_us":52,"thread_start_us":3359,"threads_started":1}
I20260812 11:28:54.091740 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.077565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37924 (local address 127.12.110.193:33293)
0812 11:28:54.081926 (+ 4361us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.081935 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.082274 (+ 339us) server_negotiation.cc:407] Connection header received
0812 11:28:54.082418 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.082425 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.082637 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.082830 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:54.083479 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.085048 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.089502 (+ 4454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.090692 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.090914 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.091078 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.091271 (+ 193us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.091421 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4188,"thread_start_us":4041,"threads_started":1}
I20260812 11:28:54.092818 12665 tablet_service.cc:1467] Processing CreateTablet for tablet c0ef03404c2046079a7a97016fe1f39b (DEFAULT_TABLE table=test-table [id=ad8da0ee04854b0ebe23991eab9c1fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:54.092818 12447 tablet_service.cc:1467] Processing CreateTablet for tablet c0ef03404c2046079a7a97016fe1f39b (DEFAULT_TABLE table=test-table [id=ad8da0ee04854b0ebe23991eab9c1fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:54.094324 12447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ef03404c2046079a7a97016fe1f39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:54.094324 12665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ef03404c2046079a7a97016fe1f39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:54.095498 12216 tablet_service.cc:1467] Processing CreateTablet for tablet c0ef03404c2046079a7a97016fe1f39b (DEFAULT_TABLE table=test-table [id=ad8da0ee04854b0ebe23991eab9c1fd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:54.097119 12216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ef03404c2046079a7a97016fe1f39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:54.109746 12816 tablet_bootstrap.cc:492] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Bootstrap starting.
I20260812 11:28:54.110615 12819 tablet_bootstrap.cc:492] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Bootstrap starting.
I20260812 11:28:54.114234 12816 tablet_bootstrap.cc:654] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:54.115234 12820 tablet_bootstrap.cc:492] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Bootstrap starting.
I20260812 11:28:54.115604 12816 log.cc:824] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:54.116079 12819 tablet_bootstrap.cc:654] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:54.117110 12819 log.cc:824] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:54.118999 12820 tablet_bootstrap.cc:654] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:54.120023 12820 log.cc:824] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:54.120775 12816 tablet_bootstrap.cc:492] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: No bootstrap required, opened a new log
I20260812 11:28:54.120787 12819 tablet_bootstrap.cc:492] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: No bootstrap required, opened a new log
I20260812 11:28:54.121176 12819 ts_tablet_manager.cc:1397] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260812 11:28:54.121176 12816 ts_tablet_manager.cc:1397] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260812 11:28:54.122678 12820 tablet_bootstrap.cc:492] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: No bootstrap required, opened a new log
I20260812 11:28:54.122960 12820 ts_tablet_manager.cc:1397] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260812 11:28:54.130650 12820 raft_consensus.cc:348] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.131040 12820 raft_consensus.cc:374] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:54.131148 12820 raft_consensus.cc:729] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2077385f81e041d2a490ab050e8f61b7, State: Initialized, Role: FOLLOWER
I20260812 11:28:54.131780 12820 consensus_queue.cc:260] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.133567 12820 ts_tablet_manager.cc:1428] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Time spent starting tablet: real 0.010s user 0.002s sys 0.007s
I20260812 11:28:54.133874 12819 raft_consensus.cc:348] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.134387 12819 raft_consensus.cc:374] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:54.134120 12816 raft_consensus.cc:348] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.134508 12819 raft_consensus.cc:729] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c60ce64583041ed9fa167b0ba12c8b4, State: Initialized, Role: FOLLOWER
I20260812 11:28:54.134593 12816 raft_consensus.cc:374] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:54.134750 12816 raft_consensus.cc:729] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d14f2f5a8814d688f7414495f86138d, State: Initialized, Role: FOLLOWER
I20260812 11:28:54.135208 12819 consensus_queue.cc:260] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.135398 12816 consensus_queue.cc:260] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.136575 12553 heartbeater.cc:503] Master 127.12.110.254:42705 was elected leader, sending a full tablet report...
I20260812 11:28:54.137570 12816 ts_tablet_manager.cc:1428] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260812 11:28:54.138610 12781 heartbeater.cc:503] Master 127.12.110.254:42705 was elected leader, sending a full tablet report...
I20260812 11:28:54.139298 12819 ts_tablet_manager.cc:1428] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Time spent starting tablet: real 0.018s user 0.016s sys 0.004s
W20260812 11:28:54.187829 12556 tablet.cc:2369] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260812 11:28:54.239949 12783 tablet.cc:2369] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:28:54.261672 12825 raft_consensus.cc:482] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:54.262049 12825 raft_consensus.cc:504] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.263988 12825 leader_election.cc:288] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c60ce64583041ed9fa167b0ba12c8b4 (127.12.110.195:40707), 8d14f2f5a8814d688f7414495f86138d (127.12.110.194:39247)
I20260812 11:28:54.272017 12834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.264454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:40707 (local address 127.12.110.193:43397)
0812 11:28:54.265428 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:54.265456 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:54.265662 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:54.266126 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:54.266131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:54.266160 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:54.266606 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.266625 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.267912 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.267918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.269735 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.269749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.271073 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.271080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.271264 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.271298 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:28:54.271416 (+ 118us) client_negotiation.cc:240] Negotiation successful
0812 11:28:54.271545 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":74,"thread_start_us":504,"threads_started":1}
I20260812 11:28:54.272078 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.264537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:43397 (local address 127.12.110.195:40707)
0812 11:28:54.264746 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.264753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.265588 (+ 835us) server_negotiation.cc:407] Connection header received
0812 11:28:54.265912 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.265918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.266003 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.266098 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:54.266801 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.267778 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.269897 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.270868 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.271051 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.271122 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.271469 (+ 347us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.271650 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:28:54.272699 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.264787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:41619 (local address 127.12.110.194:39247)
0812 11:28:54.265026 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.265033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.265770 (+ 737us) server_negotiation.cc:407] Connection header received
0812 11:28:54.266050 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.266056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.266142 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.266244 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:54.267219 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.268153 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.270279 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.271324 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.271510 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.271600 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.272244 (+ 644us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.272396 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:28:54.273483 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.264589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:39247 (local address 127.12.110.193:41619)
0812 11:28:54.265658 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:54.265676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:54.265805 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:54.266427 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:54.266434 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:54.266464 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:54.266970 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.266986 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.268342 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.268346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.270118 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:54.270133 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.271788 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.271797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:54.272029 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.272076 (+ 47us) client_negotiation.cc:769] Sending connection context
0812 11:28:54.272894 (+ 818us) client_negotiation.cc:240] Negotiation successful
0812 11:28:54.273048 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":953,"mutex_wait_us":38,"thread_start_us":659,"threads_started":1}
I20260812 11:28:54.274426 12481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ef03404c2046079a7a97016fe1f39b" candidate_uuid: "2077385f81e041d2a490ab050e8f61b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d14f2f5a8814d688f7414495f86138d" is_pre_election: true
I20260812 11:28:54.275084 12481 raft_consensus.cc:2454] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2077385f81e041d2a490ab050e8f61b7 in term 0.
I20260812 11:28:54.275707 12698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ef03404c2046079a7a97016fe1f39b" candidate_uuid: "2077385f81e041d2a490ab050e8f61b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" is_pre_election: true
I20260812 11:28:54.275871 12142 leader_election.cc:302] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2077385f81e041d2a490ab050e8f61b7, 8d14f2f5a8814d688f7414495f86138d; no voters:
I20260812 11:28:54.276261 12825 raft_consensus.cc:2781] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:28:54.276212 12698 raft_consensus.cc:2454] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2077385f81e041d2a490ab050e8f61b7 in term 0.
I20260812 11:28:54.276404 12825 raft_consensus.cc:482] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:28:54.276504 12825 raft_consensus.cc:3037] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:54.280035 12825 raft_consensus.cc:504] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.280741 12825 leader_election.cc:288] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [CANDIDATE]: Term 1 election: Requested vote from peers 1c60ce64583041ed9fa167b0ba12c8b4 (127.12.110.195:40707), 8d14f2f5a8814d688f7414495f86138d (127.12.110.194:39247)
I20260812 11:28:54.281164 12698 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ef03404c2046079a7a97016fe1f39b" candidate_uuid: "2077385f81e041d2a490ab050e8f61b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c60ce64583041ed9fa167b0ba12c8b4"
I20260812 11:28:54.281279 12481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ef03404c2046079a7a97016fe1f39b" candidate_uuid: "2077385f81e041d2a490ab050e8f61b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d14f2f5a8814d688f7414495f86138d"
I20260812 11:28:54.281378 12698 raft_consensus.cc:3037] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:54.281495 12481 raft_consensus.cc:3037] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:54.283619 12698 raft_consensus.cc:2454] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2077385f81e041d2a490ab050e8f61b7 in term 1.
I20260812 11:28:54.283721 12481 raft_consensus.cc:2454] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2077385f81e041d2a490ab050e8f61b7 in term 1.
I20260812 11:28:54.284112 12137 leader_election.cc:302] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c60ce64583041ed9fa167b0ba12c8b4, 2077385f81e041d2a490ab050e8f61b7; no voters:
I20260812 11:28:54.284425 12825 raft_consensus.cc:2781] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:54.284873 12825 raft_consensus.cc:686] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 1 LEADER]: Becoming Leader. State: Replica: 2077385f81e041d2a490ab050e8f61b7, State: Running, Role: LEADER
I20260812 11:28:54.285351 12825 consensus_queue.cc:237] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } }
I20260812 11:28:54.290005 12013 catalog_manager.cc:5495] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2077385f81e041d2a490ab050e8f61b7 (127.12.110.193). New cstate: current_term: 1 leader_uuid: "2077385f81e041d2a490ab050e8f61b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2077385f81e041d2a490ab050e8f61b7" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 33293 } health_report { overall_health: HEALTHY } } }
I20260812 11:28:54.326815 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.320082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.12.110.193:33293)
0812 11:28:54.320337 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.320342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.320543 (+ 201us) server_negotiation.cc:407] Connection header received
0812 11:28:54.320773 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.320777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.320861 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.321014 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:28:54.321688 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.322810 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.324429 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.324943 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.325051 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.325553 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:28:54.326049 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:28:54.326185 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.326375 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.326446 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:28:54.335502 12481 raft_consensus.cc:1261] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 1 FOLLOWER]: Refusing update from remote peer 2077385f81e041d2a490ab050e8f61b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:54.335505 12698 raft_consensus.cc:1261] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 1 FOLLOWER]: Refusing update from remote peer 2077385f81e041d2a490ab050e8f61b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:54.336710 12825 consensus_queue.cc:1035] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d14f2f5a8814d688f7414495f86138d" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 39247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:54.337021 12836 consensus_queue.cc:1035] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c60ce64583041ed9fa167b0ba12c8b4" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:54.348423 12841 mvcc.cc:205] Tried to move back new op lower bound from 7317643814227054592 to 7317643814035996672. Current Snapshot: MvccSnapshot[applied={T|T < 7317643814227054592}]
I20260812 11:28:54.351390 12843 mvcc.cc:205] Tried to move back new op lower bound from 7317643814227054592 to 7317643814035996672. Current Snapshot: MvccSnapshot[applied={T|T < 7317643814227054592}]
I20260812 11:28:54.362125 12012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:54.362401 12012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:54.366645 12012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3d8645ed4074056bd356970d38c1bd7: Sending DeleteTablet for 3 replicas of tablet c0ef03404c2046079a7a97016fe1f39b
I20260812 11:28:54.367996 12447 tablet_service.cc:1514] Processing DeleteTablet for tablet c0ef03404c2046079a7a97016fe1f39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:54 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:49910
I20260812 11:28:54.368093 12665 tablet_service.cc:1514] Processing DeleteTablet for tablet c0ef03404c2046079a7a97016fe1f39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:54 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:47078
I20260812 11:28:54.368407 12216 tablet_service.cc:1514] Processing DeleteTablet for tablet c0ef03404c2046079a7a97016fe1f39b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:54 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:37924
I20260812 11:28:54.369547 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 12117
I20260812 11:28:54.382398 12861 tablet_replica.cc:330] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: stopping tablet replica
I20260812 11:28:54.383059 12861 raft_consensus.cc:2229] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:54.383682 12861 raft_consensus.cc:2258] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:54.385280 12863 tablet_replica.cc:330] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: stopping tablet replica
I20260812 11:28:54.385700 12863 raft_consensus.cc:2229] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:28:54.386780 12863 raft_consensus.cc:2258] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:54.388720 12861 ts_tablet_manager.cc:1905] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:54.388727 12863 ts_tablet_manager.cc:1905] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:54.389549 12860 tablet_replica.cc:330] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: stopping tablet replica
I20260812 11:28:54.389938 12860 raft_consensus.cc:2229] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:54.390401 12860 raft_consensus.cc:2258] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:54.396039 12860 ts_tablet_manager.cc:1905] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:54.396631 12863 ts_tablet_manager.cc:1918] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:54.396816 12863 log.cc:1196] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/c0ef03404c2046079a7a97016fe1f39b
I20260812 11:28:54.397393 12863 ts_tablet_manager.cc:1939] T c0ef03404c2046079a7a97016fe1f39b P 2077385f81e041d2a490ab050e8f61b7: Deleting consensus metadata
I20260812 11:28:54.398182 12861 ts_tablet_manager.cc:1918] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:54.398363 12861 log.cc:1196] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/c0ef03404c2046079a7a97016fe1f39b
I20260812 11:28:54.398965 12861 ts_tablet_manager.cc:1939] T c0ef03404c2046079a7a97016fe1f39b P 8d14f2f5a8814d688f7414495f86138d: Deleting consensus metadata
I20260812 11:28:54.399381 11985 catalog_manager.cc:4841] TS 2077385f81e041d2a490ab050e8f61b7 (127.12.110.193:33293): tablet c0ef03404c2046079a7a97016fe1f39b (table test-table [id=ad8da0ee04854b0ebe23991eab9c1fd5]) successfully deleted
I20260812 11:28:54.400655 11991 catalog_manager.cc:4841] TS 8d14f2f5a8814d688f7414495f86138d (127.12.110.194:39247): tablet c0ef03404c2046079a7a97016fe1f39b (table test-table [id=ad8da0ee04854b0ebe23991eab9c1fd5]) successfully deleted
Aug 12 11:28:54 dist-test-slave-h6n0 krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:28:54.404174 12860 ts_tablet_manager.cc:1918] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:54.404354 12860 log.cc:1196] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/c0ef03404c2046079a7a97016fe1f39b
I20260812 11:28:54.404938 12860 ts_tablet_manager.cc:1939] T c0ef03404c2046079a7a97016fe1f39b P 1c60ce64583041ed9fa167b0ba12c8b4: Deleting consensus metadata
I20260812 11:28:54.406924 11985 catalog_manager.cc:4841] TS 1c60ce64583041ed9fa167b0ba12c8b4 (127.12.110.195:40707): tablet c0ef03404c2046079a7a97016fe1f39b (table test-table [id=ad8da0ee04854b0ebe23991eab9c1fd5]) successfully deleted
I20260812 11:28:54.411259 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.372707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37936 (local address 127.12.110.193:33293)
0812 11:28:54.372936 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.372945 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.384018 (+ 11073us) server_negotiation.cc:407] Connection header received
0812 11:28:54.384392 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.384402 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.384548 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.384652 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:54.387524 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.389125 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.391221 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.392081 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.392193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.403771 (+ 11578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:54.403786 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:54.403805 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:54.403849 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:54.408018 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.408873 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.408882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.408886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.409057 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.409501 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.409507 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.409510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.409983 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:54.410336 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.410617 (+ 281us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.410873 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:28:54.411939 12315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:54.535512 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 12117
I20260812 11:28:54.561417 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 12338
Aug 12 11:28:54 dist-test-slave-h6n0 krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:28:54.577538 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.562283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.12.110.194:39247)
0812 11:28:54.562460 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.562465 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.562484 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:28:54.562704 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.562708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.562776 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.562877 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:54.564434 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.565178 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.566447 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.566989 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.567095 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.571510 (+ 4415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:54.571522 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:54.571536 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:54.571567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:54.574704 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.575382 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.575392 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.575396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.575539 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.575930 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.575935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.575939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.576323 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:54.576643 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.576937 (+ 294us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.577176 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:28:54.578231 12546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:54.669055 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 12338
I20260812 11:28:54.697141 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 12564
Aug 12 11:28:54 dist-test-slave-h6n0 krb5kdc[11941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:28:54.713366 12804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:54.698131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.12.110.195:40707)
0812 11:28:54.698303 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:54.698307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:54.698326 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:28:54.698432 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:54.698436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:54.698513 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:54.698648 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:54.700460 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.701187 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.702849 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:54.703242 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:54.703340 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:54.707894 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:54.707908 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:54.707923 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:28:54.707956 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:54.711084 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.711670 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.711680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.711684 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.711857 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:28:54.712176 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:28:54.712179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:28:54.712181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:28:54.712511 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:54.712719 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:54.712960 (+ 241us) server_negotiation.cc:299] Negotiation successful
0812 11:28:54.713113 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:28:54.714016 12771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:54.808342 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 12564
I20260812 11:28:54.829836 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 11964
I20260812 11:28:54.830976 12067 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:54.943416 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 11964
2026-08-12T11:28:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4934 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-12T11:28:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:28:55Z Disabled control of system clock
I20260812 11:28:55.049764 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41955
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:41955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:28:55.406519 12882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:55.406847 12882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:55.406948 12882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:55.414433 12882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:28:55.414577 12882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:55.414645 12882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:55.414700 12882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:28:55.414744 12882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:28:55.423784 12882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:41955
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:55.424998 12882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:55.426769 12882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:55.439247 12893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:55.439368 12891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:55.439427 12899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:55.440711 12882 server_base.cc:1034] running on GCE node
I20260812 11:28:55.441494 12882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:55.442937 12882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:55.444176 12882 hybrid_clock.cc:648] HybridClock initialized: now 1786534135444116 us; error 51 us; skew 500 ppm
I20260812 11:28:55.444541 12882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:55.446986 12882 webserver.cc:460] Webserver started at http://127.12.110.254:43865/ using document root <none> and password file <none>
I20260812 11:28:55.447681 12882 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:55.447799 12882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:55.448164 12882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:55.451258 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "034f4ce4dae44e1880782389b47de2da"
format_stamp: "Formatted at 2026-08-12 11:28:55 on dist-test-slave-h6n0"
server_key: "61be65fecab8fc83c15bd25d427ead51"
server_key_iv: "8013b4b9e694276f5ac1d9f71d2f8d65"
server_key_version: "encryptionkey@0"
I20260812 11:28:55.452091 12882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "034f4ce4dae44e1880782389b47de2da"
format_stamp: "Formatted at 2026-08-12 11:28:55 on dist-test-slave-h6n0"
server_key: "61be65fecab8fc83c15bd25d427ead51"
server_key_iv: "8013b4b9e694276f5ac1d9f71d2f8d65"
server_key_version: "encryptionkey@0"
I20260812 11:28:55.457341 12882 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:28:55.460839 12908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:55.462373 12882 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 11:28:55.462649 12882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "034f4ce4dae44e1880782389b47de2da"
format_stamp: "Formatted at 2026-08-12 11:28:55 on dist-test-slave-h6n0"
server_key: "61be65fecab8fc83c15bd25d427ead51"
server_key_iv: "8013b4b9e694276f5ac1d9f71d2f8d65"
server_key_version: "encryptionkey@0"
I20260812 11:28:55.462877 12882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:55.478186 12882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:55.479171 12882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:55.479476 12882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:55.500240 12882 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:41955
I20260812 11:28:55.500280 12988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:41955 every 8 connection(s)
I20260812 11:28:55.501725 12882 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:28:55.505478 12989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:55.511516 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 12882
I20260812 11:28:55.511740 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:28:55.512203 12731 external_mini_cluster.cc:1428] Setting key 4b944fd4e092d6a9eb71f8776854877b
I20260812 11:28:55.516548 12989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da: Bootstrap starting.
I20260812 11:28:55.519954 12989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:55.521102 12989 log.cc:824] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:55.524163 12989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da: No bootstrap required, opened a new log
I20260812 11:28:55.527498 12994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:55.514414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.12.110.254:41955)
0812 11:28:55.515475 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:55.515486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:55.515528 (+ 42us) server_negotiation.cc:407] Connection header received
0812 11:28:55.517375 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:55.517397 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:55.517829 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:55.518157 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:55.519017 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:55.520804 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:55.522925 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:55.523605 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:55.523835 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:55.525132 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:55.525141 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:55.525155 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:55.525696 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:55.525781 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:55.525802 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:55.525937 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:55.526149 (+ 212us) server_negotiation.cc:299] Negotiation successful
0812 11:28:55.526468 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":370,"threads_started":1}
I20260812 11:28:55.532141 12989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034f4ce4dae44e1880782389b47de2da" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41955 } }
I20260812 11:28:55.532505 12989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:55.532614 12989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034f4ce4dae44e1880782389b47de2da, State: Initialized, Role: FOLLOWER
I20260812 11:28:55.533140 12989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034f4ce4dae44e1880782389b47de2da" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41955 } }
I20260812 11:28:55.533389 12989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:28:55.533541 12989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:28:55.533718 12989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:55.536223 12989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034f4ce4dae44e1880782389b47de2da" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41955 } }
I20260812 11:28:55.536731 12989 leader_election.cc:302] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 034f4ce4dae44e1880782389b47de2da; no voters:
I20260812 11:28:55.537303 12989 leader_election.cc:288] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:28:55.537526 12996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:55.538307 12996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [term 1 LEADER]: Becoming Leader. State: Replica: 034f4ce4dae44e1880782389b47de2da, State: Running, Role: LEADER
I20260812 11:28:55.538887 12996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034f4ce4dae44e1880782389b47de2da" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41955 } }
I20260812 11:28:55.539690 12989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:28:55.541574 13002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 034f4ce4dae44e1880782389b47de2da. Latest consensus state: current_term: 1 leader_uuid: "034f4ce4dae44e1880782389b47de2da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034f4ce4dae44e1880782389b47de2da" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41955 } } }
I20260812 11:28:55.542078 13002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:55.541669 12998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "034f4ce4dae44e1880782389b47de2da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034f4ce4dae44e1880782389b47de2da" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41955 } } }
I20260812 11:28:55.542359 12998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da [sys.catalog]: This master's current role is: LEADER
I20260812 11:28:55.544162 13011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:28:55.549469 13011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:28:55.557621 13011 catalog_manager.cc:1347] Generated new cluster ID: 3a72b380e99d4b9e8c7c0502121cee50
I20260812 11:28:55.557724 13011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:28:55.570366 13011 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:28:55.571815 13011 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:28:55.578437 13011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da: Generated new TSK 0
I20260812 11:28:55.579806 13011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:28:55.594723 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41955
--builtin_ntp_servers=127.12.110.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:28:55.956528 13032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:55.956844 13032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:55.957005 13032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:55.964171 13032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:55.964298 13032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:55.964485 13032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:28:55.973328 13032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:55.974681 13032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:55.976370 13032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:55.989671 13040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:55.989751 13041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:55.989777 13044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:55.991217 13032 server_base.cc:1034] running on GCE node
I20260812 11:28:55.992286 13032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:55.993383 13032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:55.994670 13032 hybrid_clock.cc:648] HybridClock initialized: now 1786534135994624 us; error 45 us; skew 500 ppm
I20260812 11:28:55.995026 13032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:55.997437 13032 webserver.cc:460] Webserver started at http://127.12.110.193:46111/ using document root <none> and password file <none>
I20260812 11:28:55.998219 13032 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:55.998309 13032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:55.998714 13032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:56.001562 13032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "ca1f18dbefb84385b425d961ea338855"
format_stamp: "Formatted at 2026-08-12 11:28:55 on dist-test-slave-h6n0"
server_key: "ff4ab2a17321f92b6c606c5b74773b60"
server_key_iv: "e5e336c79f29eb85dd2a00353b49371c"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.002362 13032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "ca1f18dbefb84385b425d961ea338855"
format_stamp: "Formatted at 2026-08-12 11:28:55 on dist-test-slave-h6n0"
server_key: "ff4ab2a17321f92b6c606c5b74773b60"
server_key_iv: "e5e336c79f29eb85dd2a00353b49371c"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.007434 13032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:28:56.010874 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.012145 13032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:28:56.012329 13032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "ca1f18dbefb84385b425d961ea338855"
format_stamp: "Formatted at 2026-08-12 11:28:55 on dist-test-slave-h6n0"
server_key: "ff4ab2a17321f92b6c606c5b74773b60"
server_key_iv: "e5e336c79f29eb85dd2a00353b49371c"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.012550 13032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:56.023839 13032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:56.024565 13032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:56.024873 13032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:56.025794 13032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:56.027323 13032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:56.027443 13032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.027578 13032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:56.027666 13032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.063880 13032 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:44191
I20260812 11:28:56.063973 13247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:44191 every 8 connection(s)
I20260812 11:28:56.065412 13032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:28:56.069337 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13032
I20260812 11:28:56.069523 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:28:56.070062 12731 external_mini_cluster.cc:1428] Setting key d560988b590bd301464a46715e5d114a
I20260812 11:28:56.073413 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41955
--builtin_ntp_servers=127.12.110.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:28:56.080397 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:56.068140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:50257 (local address 127.12.110.254:41955)
0812 11:28:56.068622 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:56.068627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:56.069769 (+ 1142us) server_negotiation.cc:407] Connection header received
0812 11:28:56.072303 (+ 2534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:56.072311 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:56.072432 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:56.072741 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:56.073794 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.074973 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.076560 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.077276 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.077424 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:56.079212 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:56.079219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:56.079223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:56.079260 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:56.079333 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:56.079350 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:56.079488 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:56.079809 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 11:28:56.079948 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":266,"threads_started":1}
I20260812 11:28:56.081568 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:56.068624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41955 (local address 127.12.110.193:50257)
0812 11:28:56.069602 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:56.069651 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:56.071777 (+ 2126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:56.072893 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:56.072902 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:56.072997 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:56.073600 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:56.073629 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.075154 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.075159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:56.076404 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:56.076416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.077598 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.077608 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:56.077850 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:56.078690 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:28:56.078697 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:56.078755 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:28:56.078853 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:56.079513 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:56.079527 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:56.079537 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:28:56.079738 (+ 201us) client_negotiation.cc:240] Negotiation successful
0812 11:28:56.080164 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":318,"threads_started":1}
I20260812 11:28:56.083242 13248 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41955
I20260812 11:28:56.083564 13248 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:56.084316 13248 heartbeater.cc:511] Master 127.12.110.254:41955 requested a full tablet report, sending...
I20260812 11:28:56.086289 12933 ts_manager.cc:194] Registered new tserver with Master: ca1f18dbefb84385b425d961ea338855 (127.12.110.193:44191)
I20260812 11:28:56.087772 12933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.193:50257
W20260812 11:28:56.425009 13257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:56.425282 13257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:56.425462 13257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:56.432672 13257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:56.432780 13257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:56.433065 13257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:28:56.442412 13257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:56.443761 13257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:56.445591 13257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:56.459379 13264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:56.459288 13267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:56.459342 13270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:56.460309 13257 server_base.cc:1034] running on GCE node
I20260812 11:28:56.460901 13257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:56.461845 13257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:56.463055 13257 hybrid_clock.cc:648] HybridClock initialized: now 1786534136463014 us; error 38 us; skew 500 ppm
I20260812 11:28:56.463375 13257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:56.465684 13257 webserver.cc:460] Webserver started at http://127.12.110.194:44091/ using document root <none> and password file <none>
I20260812 11:28:56.466403 13257 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:56.466581 13257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:56.466945 13257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:56.469808 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "c74195cbfedd4fb38943f854ede1e704"
format_stamp: "Formatted at 2026-08-12 11:28:56 on dist-test-slave-h6n0"
server_key: "98579ef3e7d5f7821413345c3a69cc05"
server_key_iv: "c1752007e8b13fd7ba1a595698293f82"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.470659 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "c74195cbfedd4fb38943f854ede1e704"
format_stamp: "Formatted at 2026-08-12 11:28:56 on dist-test-slave-h6n0"
server_key: "98579ef3e7d5f7821413345c3a69cc05"
server_key_iv: "c1752007e8b13fd7ba1a595698293f82"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.477366 13257 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 11:28:56.484097 13285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.486073 13257 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260812 11:28:56.486229 13257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "c74195cbfedd4fb38943f854ede1e704"
format_stamp: "Formatted at 2026-08-12 11:28:56 on dist-test-slave-h6n0"
server_key: "98579ef3e7d5f7821413345c3a69cc05"
server_key_iv: "c1752007e8b13fd7ba1a595698293f82"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.486411 13257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:56.499051 13257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:56.499917 13257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:56.500219 13257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:56.501207 13257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:56.502738 13257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:56.502849 13257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.502986 13257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:56.503077 13257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.538066 13257 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:36945
I20260812 11:28:56.538278 13464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:36945 every 8 connection(s)
I20260812 11:28:56.539428 13257 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:28:56.544529 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13257
I20260812 11:28:56.544713 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:28:56.545266 12731 external_mini_cluster.cc:1428] Setting key b27db4d9cdffdda83e391e761043e62f
I20260812 11:28:56.548161 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41955
--builtin_ntp_servers=127.12.110.212:33093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:28:56.552445 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:56.542133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:54385 (local address 127.12.110.254:41955)
0812 11:28:56.542346 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:56.542351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:56.543684 (+ 1333us) server_negotiation.cc:407] Connection header received
0812 11:28:56.546033 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:56.546039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:56.546130 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:56.546269 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:56.547308 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.548027 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.549631 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.550040 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.550180 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:56.551640 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:56.551651 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:56.551654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:56.551671 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:56.551703 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:56.551716 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:56.551813 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:56.552043 (+ 230us) server_negotiation.cc:299] Negotiation successful
0812 11:28:56.552142 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:28:56.553345 13469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:56.542614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41955 (local address 127.12.110.194:54385)
0812 11:28:56.543534 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:56.543583 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:56.545740 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:56.546481 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:56.546488 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:56.546628 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:56.547148 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:56.547163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.548201 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.548206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:56.549483 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:56.549495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:56.550297 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:56.550308 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:56.550520 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:56.551314 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:28:56.551317 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:56.551357 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:28:56.551432 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:56.551813 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:56.551827 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:56.551837 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:28:56.552019 (+ 182us) client_negotiation.cc:240] Negotiation successful
0812 11:28:56.552363 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":272,"threads_started":1}
I20260812 11:28:56.554849 13466 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41955
I20260812 11:28:56.555193 13466 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:56.556135 13466 heartbeater.cc:511] Master 127.12.110.254:41955 requested a full tablet report, sending...
I20260812 11:28:56.557924 12933 ts_manager.cc:194] Registered new tserver with Master: c74195cbfedd4fb38943f854ede1e704 (127.12.110.194:36945)
I20260812 11:28:56.559204 12933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.194:54385
W20260812 11:28:56.895112 13470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:56.895370 13470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:56.895454 13470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:56.902393 13470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:56.902722 13470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:56.902896 13470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:28:56.911743 13470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41955
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:56.912937 13470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:56.914635 13470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:56.927135 13483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:56.927086 13485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:56.927296 13482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:56.928071 13470 server_base.cc:1034] running on GCE node
I20260812 11:28:56.928663 13470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:56.929553 13470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:56.930779 13470 hybrid_clock.cc:648] HybridClock initialized: now 1786534136930733 us; error 40 us; skew 500 ppm
I20260812 11:28:56.931085 13470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:28:56.933245 13470 webserver.cc:460] Webserver started at http://127.12.110.195:39967/ using document root <none> and password file <none>
I20260812 11:28:56.933889 13470 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:56.934018 13470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:56.934333 13470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:56.937059 13470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "36ae61421c304074a592ddb6712d6466"
format_stamp: "Formatted at 2026-08-12 11:28:56 on dist-test-slave-h6n0"
server_key: "1e31891d813a7af38164ff7014856405"
server_key_iv: "aa1d1dfd63d65e2a767800cce0eb6e6c"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.937829 13470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "36ae61421c304074a592ddb6712d6466"
format_stamp: "Formatted at 2026-08-12 11:28:56 on dist-test-slave-h6n0"
server_key: "1e31891d813a7af38164ff7014856405"
server_key_iv: "aa1d1dfd63d65e2a767800cce0eb6e6c"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.943007 13470 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:28:56.946406 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.947649 13470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:28:56.947824 13470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "36ae61421c304074a592ddb6712d6466"
format_stamp: "Formatted at 2026-08-12 11:28:56 on dist-test-slave-h6n0"
server_key: "1e31891d813a7af38164ff7014856405"
server_key_iv: "aa1d1dfd63d65e2a767800cce0eb6e6c"
server_key_version: "encryptionkey@0"
I20260812 11:28:56.948031 13470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:56.963379 13470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:56.964138 13470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:56.964445 13470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:56.965533 13470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:28:56.967389 13470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:28:56.967531 13470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:56.967661 13470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:28:56.967774 13470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:57.005653 13470 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:40705
I20260812 11:28:57.005782 13687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:40705 every 8 connection(s)
I20260812 11:28:57.007050 13470 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:28:57.014775 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13470
I20260812 11:28:57.015024 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:28:57.015599 12731 external_mini_cluster.cc:1428] Setting key 341ba337ab1050d9ab4ed55a3eaf4e2f
I20260812 11:28:57.020504 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.009868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:57065 (local address 127.12.110.254:41955)
0812 11:28:57.010070 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.010074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.011459 (+ 1385us) server_negotiation.cc:407] Connection header received
0812 11:28:57.013255 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.013259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.013332 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.013447 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:57.014660 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.015571 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.017613 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.018064 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.018202 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.019662 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:57.019667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:57.019671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:57.019687 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:57.019717 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:57.019729 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:57.019836 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.020080 (+ 244us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.020172 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:28:57.021541 13690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.010313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41955 (local address 127.12.110.195:57065)
0812 11:28:57.011280 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.011336 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:57.012978 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:57.013665 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:57.013672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:57.013752 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:28:57.014428 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.014457 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.015744 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.015756 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.017473 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.017486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.018286 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.018294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.018609 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.019337 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:28:57.019340 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:28:57.019377 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:28:57.019452 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:28:57.019836 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:28:57.019848 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:28:57.019858 (+ 10us) client_negotiation.cc:769] Sending connection context
0812 11:28:57.020049 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 11:28:57.020410 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":341,"threads_started":1}
I20260812 11:28:57.022918 13688 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41955
I20260812 11:28:57.023236 13688 heartbeater.cc:464] Registering TS with master...
I20260812 11:28:57.024127 13688 heartbeater.cc:511] Master 127.12.110.254:41955 requested a full tablet report, sending...
I20260812 11:28:57.025528 12933 ts_manager.cc:194] Registered new tserver with Master: 36ae61421c304074a592ddb6712d6466 (127.12.110.195:40705)
I20260812 11:28:57.026324 12933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.195:57065
I20260812 11:28:57.030973 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:28:57.041538 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.035482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33970 (local address 127.12.110.254:41955)
0812 11:28:57.035776 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.035783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.036078 (+ 295us) server_negotiation.cc:407] Connection header received
0812 11:28:57.036277 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.036287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.036365 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.036484 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:57.036949 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.037658 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.039078 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.039510 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.039651 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.040800 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:57.040805 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:57.040808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:57.040825 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:57.040854 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:57.040865 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:57.041001 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.041204 (+ 203us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.041258 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:28:57.046345 12933 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:28:57.053580 12933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:28:57.089955 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.066431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:36945 (local address 127.0.0.1:37686)
0812 11:28:57.080860 (+ 14429us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.080931 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:57.081110 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:57.082828 (+ 1718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:57.082839 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:57.082873 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:57.083443 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.083456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.085481 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.085487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.087482 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.087498 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.088929 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.088937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.089138 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.089199 (+ 61us) client_negotiation.cc:769] Sending connection context
0812 11:28:57.089312 (+ 113us) client_negotiation.cc:240] Negotiation successful
0812 11:28:57.089487 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14141,"mutex_wait_us":2,"thread_start_us":16095,"threads_started":1}
I20260812 11:28:57.090150 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.067102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37686 (local address 127.12.110.194:36945)
0812 11:28:57.076815 (+ 9713us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.076830 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.081314 (+ 4484us) server_negotiation.cc:407] Connection header received
0812 11:28:57.081474 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.081485 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.081794 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.082027 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:57.083651 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.085292 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.087676 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.088746 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.089056 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.089500 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.089609 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.089781 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9526,"thread_start_us":9394,"threads_started":1}
I20260812 11:28:57.091352 13713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.067635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51134)
0812 11:28:57.083143 (+ 15508us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.083167 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:57.083304 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:57.084033 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:57.084040 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:57.084069 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:57.084521 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.084532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.086392 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.086402 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.088598 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.088608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.090588 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.090596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.090764 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.090794 (+ 30us) client_negotiation.cc:769] Sending connection context
0812 11:28:57.090880 (+ 86us) client_negotiation.cc:240] Negotiation successful
0812 11:28:57.091016 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15374,"mutex_wait_us":6,"thread_start_us":15264,"threads_started":1}
I20260812 11:28:57.091831 13706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.066638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:40705 (local address 127.0.0.1:34288)
0812 11:28:57.082822 (+ 16184us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.082855 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:57.083022 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:57.083666 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:57.083670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:57.083694 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:57.084198 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.084219 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.086000 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.086008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.088317 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.088332 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.090929 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.090935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.091110 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.091207 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 11:28:57.091300 (+ 93us) client_negotiation.cc:240] Negotiation successful
0812 11:28:57.091447 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16040,"mutex_wait_us":28,"thread_start_us":14063,"threads_started":1}
I20260812 11:28:57.092427 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.067147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.12.110.195:40705)
0812 11:28:57.082842 (+ 15695us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.082852 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.083158 (+ 306us) server_negotiation.cc:407] Connection header received
0812 11:28:57.083272 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.083281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.083485 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.083685 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:57.084391 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.085854 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.088551 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.090391 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.090607 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.090794 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.091906 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.092084 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15514,"thread_start_us":15335,"threads_started":1}
I20260812 11:28:57.092464 13248 heartbeater.cc:503] Master 127.12.110.254:41955 was elected leader, sending a full tablet report...
I20260812 11:28:57.093276 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.067865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.12.110.193:44191)
0812 11:28:57.080106 (+ 12241us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.080128 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.083304 (+ 3176us) server_negotiation.cc:407] Connection header received
0812 11:28:57.083459 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.083467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.083697 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.083917 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:57.084725 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.086227 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.088768 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.090313 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.092468 (+ 2155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.092696 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.092768 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.092939 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12060,"thread_start_us":12260,"threads_started":1}
I20260812 11:28:57.094779 13358 tablet_service.cc:1467] Processing CreateTablet for tablet 1a44aab8582f4e77b249c45448907180 (DEFAULT_TABLE table=test-table [id=dd5ebb182de44fbcbf4acf0f8e4aabd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:57.095995 13358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a44aab8582f4e77b249c45448907180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:57.097007 13573 tablet_service.cc:1467] Processing CreateTablet for tablet 1a44aab8582f4e77b249c45448907180 (DEFAULT_TABLE table=test-table [id=dd5ebb182de44fbcbf4acf0f8e4aabd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:57.097005 13137 tablet_service.cc:1467] Processing CreateTablet for tablet 1a44aab8582f4e77b249c45448907180 (DEFAULT_TABLE table=test-table [id=dd5ebb182de44fbcbf4acf0f8e4aabd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:28:57.098425 13137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a44aab8582f4e77b249c45448907180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:57.098450 13573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a44aab8582f4e77b249c45448907180. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:57.108582 13715 tablet_bootstrap.cc:492] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Bootstrap starting.
I20260812 11:28:57.112601 13715 tablet_bootstrap.cc:654] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:57.113325 13718 tablet_bootstrap.cc:492] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Bootstrap starting.
I20260812 11:28:57.114080 13715 log.cc:824] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:57.114185 13716 tablet_bootstrap.cc:492] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: Bootstrap starting.
I20260812 11:28:57.117115 13715 tablet_bootstrap.cc:492] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: No bootstrap required, opened a new log
I20260812 11:28:57.117412 13715 ts_tablet_manager.cc:1397] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20260812 11:28:57.117846 13718 tablet_bootstrap.cc:654] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:57.119202 13716 tablet_bootstrap.cc:654] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: Neither blocks nor log segments found. Creating new log.
I20260812 11:28:57.119235 13718 log.cc:824] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:57.120313 13716 log.cc:824] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: Log is configured to *not* fsync() on all Append() calls
I20260812 11:28:57.122268 13718 tablet_bootstrap.cc:492] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: No bootstrap required, opened a new log
I20260812 11:28:57.122627 13718 ts_tablet_manager.cc:1397] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260812 11:28:57.123404 13716 tablet_bootstrap.cc:492] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: No bootstrap required, opened a new log
I20260812 11:28:57.123663 13716 ts_tablet_manager.cc:1397] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260812 11:28:57.126451 13715 raft_consensus.cc:348] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.126926 13715 raft_consensus.cc:374] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:57.127059 13715 raft_consensus.cc:729] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c74195cbfedd4fb38943f854ede1e704, State: Initialized, Role: FOLLOWER
I20260812 11:28:57.127679 13715 consensus_queue.cc:260] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.128805 13466 heartbeater.cc:503] Master 127.12.110.254:41955 was elected leader, sending a full tablet report...
I20260812 11:28:57.129344 13715 ts_tablet_manager.cc:1428] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260812 11:28:57.134076 13716 raft_consensus.cc:348] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.134603 13716 raft_consensus.cc:374] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:57.134742 13716 raft_consensus.cc:729] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca1f18dbefb84385b425d961ea338855, State: Initialized, Role: FOLLOWER
I20260812 11:28:57.135355 13716 consensus_queue.cc:260] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.135854 13718 raft_consensus.cc:348] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.136309 13718 raft_consensus.cc:374] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:28:57.136423 13718 raft_consensus.cc:729] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36ae61421c304074a592ddb6712d6466, State: Initialized, Role: FOLLOWER
I20260812 11:28:57.136976 13718 consensus_queue.cc:260] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.138046 13688 heartbeater.cc:503] Master 127.12.110.254:41955 was elected leader, sending a full tablet report...
I20260812 11:28:57.138339 13716 ts_tablet_manager.cc:1428] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260812 11:28:57.138675 13718 ts_tablet_manager.cc:1428] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Time spent starting tablet: real 0.016s user 0.012s sys 0.001s
I20260812 11:28:57.148187 13723 raft_consensus.cc:482] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:28:57.148566 13723 raft_consensus.cc:504] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.150298 13723 leader_election.cc:288] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36ae61421c304074a592ddb6712d6466 (127.12.110.195:40705), ca1f18dbefb84385b425d961ea338855 (127.12.110.193:44191)
I20260812 11:28:57.157624 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.150824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:52325 (local address 127.12.110.193:44191)
0812 11:28:57.151064 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.151072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.152020 (+ 948us) server_negotiation.cc:407] Connection header received
0812 11:28:57.152297 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.152301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.152400 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.152498 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:57.153146 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.153800 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.155899 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.156692 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.156881 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.156956 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.157260 (+ 304us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.157390 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260812 11:28:57.157646 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.150813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:35645 (local address 127.12.110.195:40705)
0812 11:28:57.151028 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.151052 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.151957 (+ 905us) server_negotiation.cc:407] Connection header received
0812 11:28:57.152201 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.152206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.152285 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.152380 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:28:57.152988 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.153775 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.155866 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.156645 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.156765 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.156861 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.157297 (+ 436us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.157415 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:28:57.157737 13729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.150828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.12.110.194:52325)
0812 11:28:57.151878 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.151894 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:57.152103 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:57.152516 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:57.152526 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:57.152561 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:57.152979 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.152990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.153959 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.153966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.155683 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.155696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.156942 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.156947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.157094 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.157126 (+ 32us) client_negotiation.cc:769] Sending connection context
0812 11:28:57.157199 (+ 73us) client_negotiation.cc:240] Negotiation successful
0812 11:28:57.157348 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"mutex_wait_us":91,"thread_start_us":429,"threads_started":1}
I20260812 11:28:57.157764 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.150755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:40705 (local address 127.12.110.194:35645)
0812 11:28:57.151839 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.151856 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0812 11:28:57.152014 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:28:57.152398 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:28:57.152403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:28:57.152430 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:28:57.152837 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.152851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.153909 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.153914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.155667 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:28:57.155679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.156862 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.156869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:28:57.157105 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.157139 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:28:57.157248 (+ 109us) client_negotiation.cc:240] Negotiation successful
0812 11:28:57.157372 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":133,"thread_start_us":708,"threads_started":1}
I20260812 11:28:57.158806 13607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a44aab8582f4e77b249c45448907180" candidate_uuid: "c74195cbfedd4fb38943f854ede1e704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ae61421c304074a592ddb6712d6466" is_pre_election: true
I20260812 11:28:57.159179 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a44aab8582f4e77b249c45448907180" candidate_uuid: "c74195cbfedd4fb38943f854ede1e704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1f18dbefb84385b425d961ea338855" is_pre_election: true
I20260812 11:28:57.159451 13607 raft_consensus.cc:2454] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74195cbfedd4fb38943f854ede1e704 in term 0.
I20260812 11:28:57.159727 13168 raft_consensus.cc:2454] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c74195cbfedd4fb38943f854ede1e704 in term 0.
I20260812 11:28:57.160228 13291 leader_election.cc:302] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ae61421c304074a592ddb6712d6466, c74195cbfedd4fb38943f854ede1e704; no voters:
I20260812 11:28:57.160658 13723 raft_consensus.cc:2781] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:28:57.160817 13723 raft_consensus.cc:482] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:28:57.160943 13723 raft_consensus.cc:3037] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:57.163271 13723 raft_consensus.cc:504] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.163872 13723 leader_election.cc:288] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [CANDIDATE]: Term 1 election: Requested vote from peers 36ae61421c304074a592ddb6712d6466 (127.12.110.195:40705), ca1f18dbefb84385b425d961ea338855 (127.12.110.193:44191)
I20260812 11:28:57.164378 13607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a44aab8582f4e77b249c45448907180" candidate_uuid: "c74195cbfedd4fb38943f854ede1e704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36ae61421c304074a592ddb6712d6466"
I20260812 11:28:57.164378 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a44aab8582f4e77b249c45448907180" candidate_uuid: "c74195cbfedd4fb38943f854ede1e704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca1f18dbefb84385b425d961ea338855"
I20260812 11:28:57.164592 13607 raft_consensus.cc:3037] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:57.164590 13168 raft_consensus.cc:3037] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:28:57.167905 13168 raft_consensus.cc:2454] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74195cbfedd4fb38943f854ede1e704 in term 1.
I20260812 11:28:57.167905 13607 raft_consensus.cc:2454] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c74195cbfedd4fb38943f854ede1e704 in term 1.
I20260812 11:28:57.168309 13291 leader_election.cc:302] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36ae61421c304074a592ddb6712d6466, c74195cbfedd4fb38943f854ede1e704; no voters:
I20260812 11:28:57.168623 13723 raft_consensus.cc:2781] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:28:57.169190 13723 raft_consensus.cc:686] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 1 LEADER]: Becoming Leader. State: Replica: c74195cbfedd4fb38943f854ede1e704, State: Running, Role: LEADER
I20260812 11:28:57.169656 13723 consensus_queue.cc:237] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } }
I20260812 11:28:57.174490 12932 catalog_manager.cc:5495] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 reported cstate change: term changed from 0 to 1, leader changed from <none> to c74195cbfedd4fb38943f854ede1e704 (127.12.110.194). New cstate: current_term: 1 leader_uuid: "c74195cbfedd4fb38943f854ede1e704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c74195cbfedd4fb38943f854ede1e704" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 36945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 } health_report { overall_health: UNKNOWN } } }
I20260812 11:28:57.206791 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.201073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.12.110.194:36945)
0812 11:28:57.201330 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.201337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.201448 (+ 111us) server_negotiation.cc:407] Connection header received
0812 11:28:57.201697 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.201701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.201794 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.201963 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:28:57.202342 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.203132 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.204528 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.204946 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.205085 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.205668 (+ 583us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:28:57.206125 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:28:57.206318 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.206479 (+ 161us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.206560 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260812 11:28:57.214850 13168 raft_consensus.cc:1261] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 1 FOLLOWER]: Refusing update from remote peer c74195cbfedd4fb38943f854ede1e704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:57.214860 13607 raft_consensus.cc:1261] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 1 FOLLOWER]: Refusing update from remote peer c74195cbfedd4fb38943f854ede1e704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:28:57.215804 13733 consensus_queue.cc:1035] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36ae61421c304074a592ddb6712d6466" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 40705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:57.216230 13723 consensus_queue.cc:1035] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca1f18dbefb84385b425d961ea338855" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 44191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:28:57.228202 13737 mvcc.cc:205] Tried to move back new op lower bound from 7317643826021224448 to 7317643825850900480. Current Snapshot: MvccSnapshot[applied={T|T < 7317643826021224448}]
I20260812 11:28:57.248251 12932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:57.248978 12932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:28:57.252830 12932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 034f4ce4dae44e1880782389b47de2da: Sending DeleteTablet for 3 replicas of tablet 1a44aab8582f4e77b249c45448907180
I20260812 11:28:57.253965 13573 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a44aab8582f4e77b249c45448907180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:57 UTC) from {username='slave'} at 127.0.0.1:34288
I20260812 11:28:57.253960 13358 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a44aab8582f4e77b249c45448907180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:57 UTC) from {username='slave'} at 127.0.0.1:37686
I20260812 11:28:57.255108 13137 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a44aab8582f4e77b249c45448907180 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:28:57 UTC) from {username='slave'} at 127.0.0.1:51134
I20260812 11:28:57.255424 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13032
I20260812 11:28:57.275103 13757 tablet_replica.cc:330] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: stopping tablet replica
I20260812 11:28:57.275578 13757 raft_consensus.cc:2229] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:57.276094 13757 raft_consensus.cc:2258] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:57.276907 13755 tablet_replica.cc:330] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: stopping tablet replica
I20260812 11:28:57.277294 13755 raft_consensus.cc:2229] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:28:57.278394 13757 ts_tablet_manager.cc:1905] T 1a44aab8582f4e77b249c45448907180 P ca1f18dbefb84385b425d961ea338855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:57.278581 13754 tablet_replica.cc:330] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: stopping tablet replica
I20260812 11:28:57.278964 13754 raft_consensus.cc:2229] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:28:57.279426 13754 raft_consensus.cc:2258] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:57.279896 13755 raft_consensus.cc:2258] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:28:57.280484 13714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.256650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.12.110.193:44191)
0812 11:28:57.256815 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.256822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.267316 (+ 10494us) server_negotiation.cc:407] Connection header received
0812 11:28:57.267847 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.267853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.267972 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.268052 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:57.268729 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.269919 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.271884 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.272564 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.272662 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.278180 (+ 5518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:57.278191 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:57.278206 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:57.279067 (+ 861us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:57.279134 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:57.279161 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:57.279287 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.279927 (+ 640us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.280071 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:28:57.281388 13237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:57.282068 13755 ts_tablet_manager.cc:1905] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:57.283680 13754 ts_tablet_manager.cc:1905] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:28:57.289150 13754 ts_tablet_manager.cc:1918] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:57.289340 13754 log.cc:1196] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/1a44aab8582f4e77b249c45448907180
I20260812 11:28:57.289932 13755 ts_tablet_manager.cc:1918] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:28:57.289978 13754 ts_tablet_manager.cc:1939] T 1a44aab8582f4e77b249c45448907180 P 36ae61421c304074a592ddb6712d6466: Deleting consensus metadata
I20260812 11:28:57.290108 13755 log.cc:1196] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/1a44aab8582f4e77b249c45448907180
I20260812 11:28:57.290721 13755 ts_tablet_manager.cc:1939] T 1a44aab8582f4e77b249c45448907180 P c74195cbfedd4fb38943f854ede1e704: Deleting consensus metadata
I20260812 11:28:57.291914 12912 catalog_manager.cc:4841] TS 36ae61421c304074a592ddb6712d6466 (127.12.110.195:40705): tablet 1a44aab8582f4e77b249c45448907180 (table test-table [id=dd5ebb182de44fbcbf4acf0f8e4aabd4]) successfully deleted
I20260812 11:28:57.292277 12909 catalog_manager.cc:4841] TS c74195cbfedd4fb38943f854ede1e704 (127.12.110.194:36945): tablet 1a44aab8582f4e77b249c45448907180 (table test-table [id=dd5ebb182de44fbcbf4acf0f8e4aabd4]) successfully deleted
I20260812 11:28:57.377946 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13032
W20260812 11:28:57.414652 12911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.110.193:44191 (error 108)
I20260812 11:28:57.415376 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13257
I20260812 11:28:57.416882 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.416317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51152)
0812 11:28:57.416563 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.416702 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
W20260812 11:28:57.417225 12911 catalog_manager.cc:4584] TS ca1f18dbefb84385b425d961ea338855 (127.12.110.193:44191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a44aab8582f4e77b249c45448907180: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
I20260812 11:28:57.423786 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.416209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.12.110.194:36945)
0812 11:28:57.416402 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.416409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.416510 (+ 101us) server_negotiation.cc:407] Connection header received
0812 11:28:57.416721 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.416727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.416832 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.416957 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:57.417511 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.418797 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.420108 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.420895 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.421046 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.422029 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:57.422040 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:57.422059 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:57.422880 (+ 821us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:57.422947 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:57.422971 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:57.423087 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.423308 (+ 221us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.423405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:28:57.424595 13458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:57.471062 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.470482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51166)
0812 11:28:57.470727 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.470859 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260812 11:28:57.472386 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.471818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51172)
0812 11:28:57.472091 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.472195 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260812 11:28:57.533099 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.532382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51174)
0812 11:28:57.532743 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.532869 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260812 11:28:57.534335 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.533900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51190)
0812 11:28:57.534079 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.534175 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 11:28:57.541934 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13257
I20260812 11:28:57.562927 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13470
I20260812 11:28:57.570057 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.563744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34294 (local address 127.12.110.195:40705)
0812 11:28:57.563962 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:57.563969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:57.564077 (+ 108us) server_negotiation.cc:407] Connection header received
0812 11:28:57.564324 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:57.564329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:57.564414 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:57.564539 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:57.564936 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.565691 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.567100 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:57.567546 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:28:57.567678 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:28:57.568757 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:28:57.568764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:28:57.568778 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:28:57.569318 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:28:57.569366 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:28:57.569385 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:28:57.569517 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0812 11:28:57.569650 (+ 133us) server_negotiation.cc:299] Negotiation successful
0812 11:28:57.569753 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260812 11:28:57.570927 13680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:57.615043 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.614502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51202)
0812 11:28:57.614761 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.614874 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260812 11:28:57.616156 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.615784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51212)
0812 11:28:57.615987 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.616054 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260812 11:28:57.659736 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13470
I20260812 11:28:57.683444 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 12882
I20260812 11:28:57.684800 12983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:28:57.798511 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:57.797838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:44191 (local address 127.0.0.1:51214)
0812 11:28:57.798095 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0812 11:28:57.798228 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:44191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260812 11:28:57.798651 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 12882
2026-08-12T11:28:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2853 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:28:57 dist-test-slave-h6n0 krb5kdc[13768](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:28:57 dist-test-slave-h6n0 krb5kdc[13768](info): set up 2 sockets
Aug 12 11:28:57 dist-test-slave-h6n0 krb5kdc[13768](info): commencing operation
krb5kdc: starting...
W20260812 11:28:59.202418 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:28:59 dist-test-slave-h6n0 krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:28:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:28:59.527822 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37531
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:37531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:28:59.873306 13799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:28:59.873593 13799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:28:59.873771 13799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:28:59.880827 13799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:28:59.880925 13799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:28:59.881026 13799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:28:59.881126 13799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:28:59.881211 13799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:28:59.891283 13799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:37531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:37531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:28:59.892709 13799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:28:59.894088 13799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:28:59.906569 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:59.906613 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:28:59.906648 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:28:59.907943 13799 server_base.cc:1034] running on GCE node
I20260812 11:28:59.909085 13799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:28:59.910737 13799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:28:59.911993 13799 hybrid_clock.cc:648] HybridClock initialized: now 1786534139911935 us; error 38 us; skew 500 ppm
I20260812 11:28:59.912412 13799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:28:59 dist-test-slave-h6n0 krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534139, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:28:59.916075 13799 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:28:59.917654 13799 webserver.cc:460] Webserver started at http://127.12.110.254:35445/ using document root <none> and password file <none>
I20260812 11:28:59.918340 13799 fs_manager.cc:359] Metadata directory not provided
I20260812 11:28:59.918489 13799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:28:59.918864 13799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:28:59.921952 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "2c685e70067c4e3a9e70e0fab0104443"
format_stamp: "Formatted at 2026-08-12 11:28:59 on dist-test-slave-h6n0"
server_key: "d245f6cc3912bec758cb2e5367572892"
server_key_iv: "82eabc66cd6c25efa29055985412aa30"
server_key_version: "encryptionkey@0"
I20260812 11:28:59.922821 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "2c685e70067c4e3a9e70e0fab0104443"
format_stamp: "Formatted at 2026-08-12 11:28:59 on dist-test-slave-h6n0"
server_key: "d245f6cc3912bec758cb2e5367572892"
server_key_iv: "82eabc66cd6c25efa29055985412aa30"
server_key_version: "encryptionkey@0"
I20260812 11:28:59.928566 13799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260812 11:28:59.932588 13825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:28:59.933921 13799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:28:59.934077 13799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "2c685e70067c4e3a9e70e0fab0104443"
format_stamp: "Formatted at 2026-08-12 11:28:59 on dist-test-slave-h6n0"
server_key: "d245f6cc3912bec758cb2e5367572892"
server_key_iv: "82eabc66cd6c25efa29055985412aa30"
server_key_version: "encryptionkey@0"
I20260812 11:28:59.934317 13799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:28:59.958828 13799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:28:59.964720 13799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:28:59.965098 13799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:28:59.985165 13799 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:37531
I20260812 11:28:59.985188 13922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:37531 every 8 connection(s)
I20260812 11:28:59.986521 13799 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:28:59.989800 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13799
I20260812 11:28:59.989991 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:28:59.990263 13923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:28:59.990497 12731 external_mini_cluster.cc:1428] Setting key f86fdce6133894ed72e104794d7d02b8
I20260812 11:29:00.004663 13923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443: Bootstrap starting.
Aug 12 11:29:00 dist-test-slave-h6n0 krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:00.007822 13923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:00.008841 13923 log.cc:824] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:00.011576 13923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443: No bootstrap required, opened a new log
I20260812 11:29:00.013793 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:28:59.992334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39772 (local address 127.12.110.254:37531)
0812 11:28:59.993158 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0812 11:28:59.993169 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:28:59.993206 (+ 37us) server_negotiation.cc:407] Connection header received
0812 11:28:59.994899 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:28:59.994919 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:28:59.995266 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:28:59.995571 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:28:59.997473 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:28:59.998825 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:00.000644 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:00.001300 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:00.001457 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:00.006425 (+ 4968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:00.006433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:00.006446 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:00.006480 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:00.009866 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:00.010584 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:00.010598 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:00.010604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:00.010761 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:00.011172 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:00.011177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:00.011182 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:00.011728 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:00.012049 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:00.012484 (+ 435us) server_negotiation.cc:299] Negotiation successful
0812 11:29:00.012866 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":368,"threads_started":1}
I20260812 11:29:00.019232 13923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c685e70067c4e3a9e70e0fab0104443" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37531 } }
I20260812 11:29:00.019577 13923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:00.019704 13923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c685e70067c4e3a9e70e0fab0104443, State: Initialized, Role: FOLLOWER
I20260812 11:29:00.020234 13923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c685e70067c4e3a9e70e0fab0104443" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37531 } }
I20260812 11:29:00.020471 13923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:00.020627 13923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:00.020823 13923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:00.023321 13923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c685e70067c4e3a9e70e0fab0104443" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37531 } }
I20260812 11:29:00.023813 13923 leader_election.cc:302] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c685e70067c4e3a9e70e0fab0104443; no voters:
I20260812 11:29:00.024379 13923 leader_election.cc:288] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:00.024533 13934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:00.025184 13934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [term 1 LEADER]: Becoming Leader. State: Replica: 2c685e70067c4e3a9e70e0fab0104443, State: Running, Role: LEADER
I20260812 11:29:00.025712 13934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c685e70067c4e3a9e70e0fab0104443" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37531 } }
I20260812 11:29:00.026309 13923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:00.028488 13938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c685e70067c4e3a9e70e0fab0104443. Latest consensus state: current_term: 1 leader_uuid: "2c685e70067c4e3a9e70e0fab0104443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c685e70067c4e3a9e70e0fab0104443" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37531 } } }
I20260812 11:29:00.028498 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c685e70067c4e3a9e70e0fab0104443" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c685e70067c4e3a9e70e0fab0104443" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 37531 } } }
I20260812 11:29:00.028923 13938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:00.028923 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:00.030308 13944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:00.036008 13944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:00.044428 13944 catalog_manager.cc:1347] Generated new cluster ID: eadf316a0d8344e983834a93eddb81f7
I20260812 11:29:00.044544 13944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:00.061852 13944 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:00.063182 13944 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:00.070007 13944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443: Generated new TSK 0
I20260812 11:29:00.071062 13944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.12.110.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
Entry for principal kudu/127.12.110.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab.
I20260812 11:29:00.155784 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37531
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:00.543502 13966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:00.543771 13966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:00.543871 13966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:00.551016 13966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:00.551139 13966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:00.551321 13966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:29:00.561700 13966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.193.keytab
--principal=kudu/127.12.110.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:00.563138 13966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:00.564489 13966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:00.577893 13980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:00.577888 13978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:00.578557 13983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:00.578851 13966 server_base.cc:1034] running on GCE node
I20260812 11:29:00.579497 13966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:00.580363 13966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:00.581598 13966 hybrid_clock.cc:648] HybridClock initialized: now 1786534140581553 us; error 40 us; skew 500 ppm
I20260812 11:29:00.581914 13966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:00 dist-test-slave-h6n0 krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534140, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:00.585632 13966 init.cc:377] Logged in from keytab as kudu/127.12.110.193@KRBTEST.COM (short username kudu)
I20260812 11:29:00.588339 13966 webserver.cc:460] Webserver started at http://127.12.110.193:38997/ using document root <none> and password file <none>
I20260812 11:29:00.589304 13966 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:00.589507 13966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:00.590041 13966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:00.593250 13966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "cae887b58cad4f408c93fda9148f0a60"
format_stamp: "Formatted at 2026-08-12 11:29:00 on dist-test-slave-h6n0"
server_key: "84b4634884ce293c0a91f5bc4e845c85"
server_key_iv: "72bca0b567304a12204715a48e28dfb8"
server_key_version: "encryptionkey@0"
I20260812 11:29:00.594087 13966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "cae887b58cad4f408c93fda9148f0a60"
format_stamp: "Formatted at 2026-08-12 11:29:00 on dist-test-slave-h6n0"
server_key: "84b4634884ce293c0a91f5bc4e845c85"
server_key_iv: "72bca0b567304a12204715a48e28dfb8"
server_key_version: "encryptionkey@0"
I20260812 11:29:00.599421 13966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260812 11:29:00.603060 13998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:00.604705 13966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260812 11:29:00.604918 13966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "cae887b58cad4f408c93fda9148f0a60"
format_stamp: "Formatted at 2026-08-12 11:29:00 on dist-test-slave-h6n0"
server_key: "84b4634884ce293c0a91f5bc4e845c85"
server_key_iv: "72bca0b567304a12204715a48e28dfb8"
server_key_version: "encryptionkey@0"
I20260812 11:29:00.605134 13966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:00.629916 13966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:00.634673 13966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:00.634975 13966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:00.635977 13966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:29:00.637353 13966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:29:00.637461 13966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:00.637629 13966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:29:00.637708 13966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:00.674156 13966 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:35579
I20260812 11:29:00.674180 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:35579 every 8 connection(s)
I20260812 11:29:00.675490 13966 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:29:00.684489 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 13966
I20260812 11:29:00.684705 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:29:00.685158 12731 external_mini_cluster.cc:1428] Setting key ae9e4962aee4031620bbdf9664ae76af
Aug 12 11:29:00 dist-test-slave-h6n0 krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534140, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.193@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:00.695603 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:00.678394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:35987 (local address 127.12.110.254:37531)
0812 11:29:00.678952 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:00.678959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:00.679747 (+ 788us) server_negotiation.cc:407] Connection header received
0812 11:29:00.681585 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:00.681589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:00.681669 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:00.681771 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:00.683254 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:00.683983 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:00.685697 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:00.686107 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:00.686256 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:00.689782 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:00.689787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:00.689790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:00.689823 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:00.692440 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:00.693184 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:00.693189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:00.693192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:00.693314 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:00.693774 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:00.693778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:00.693780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:00.694069 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:00.694232 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:00.695016 (+ 784us) server_negotiation.cc:299] Negotiation successful
0812 11:29:00.695247 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":281,"threads_started":1}
I20260812 11:29:00.696544 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:00.678734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37531 (local address 127.12.110.193:35987)
0812 11:29:00.679566 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:00.679617 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:00.681315 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:00.681971 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:00.681978 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:00.682521 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:00.683085 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:00.683102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:00.684139 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:00.684143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:00.685545 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:00.685558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:00.686330 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:00.686335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:00.686602 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:00.687329 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:00.687351 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:00.689557 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:00.692590 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:00.692599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:00.692614 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:00.693045 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:00.693435 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:00.693440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:00.693444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:00.693640 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:00.694213 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:00.694221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:00.694614 (+ 393us) client_negotiation.cc:769] Sending connection context
0812 11:29:00.694890 (+ 276us) client_negotiation.cc:240] Negotiation successful
0812 11:29:00.695430 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":354,"threads_started":1}
I20260812 11:29:00.698066 14193 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37531
I20260812 11:29:00.698401 14193 heartbeater.cc:464] Registering TS with master...
I20260812 11:29:00.699151 14193 heartbeater.cc:511] Master 127.12.110.254:37531 requested a full tablet report, sending...
I20260812 11:29:00.701354 13859 ts_manager.cc:194] Registered new tserver with Master: cae887b58cad4f408c93fda9148f0a60 (127.12.110.193:35579)
I20260812 11:29:00.702977 13859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.193@KRBTEST.COM'} at 127.12.110.193:35987
WARNING: no policy specified for kudu/127.12.110.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
Entry for principal kudu/127.12.110.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab.
I20260812 11:29:00.759897 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37531
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:01.108045 14202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:01.108428 14202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:01.108644 14202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:01.116698 14202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:01.116896 14202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:01.117141 14202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:29:01.128791 14202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.194.keytab
--principal=kudu/127.12.110.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:01.130184 14202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:01.131619 14202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:01.146209 14211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:01.146250 14214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:01.146294 14210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:01.147336 14202 server_base.cc:1034] running on GCE node
I20260812 11:29:01.148000 14202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:01.148936 14202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:01.150441 14202 hybrid_clock.cc:648] HybridClock initialized: now 1786534141150395 us; error 32 us; skew 500 ppm
I20260812 11:29:01.150823 14202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:01 dist-test-slave-h6n0 krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534141, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:01.154798 14202 init.cc:377] Logged in from keytab as kudu/127.12.110.194@KRBTEST.COM (short username kudu)
I20260812 11:29:01.156817 14202 webserver.cc:460] Webserver started at http://127.12.110.194:41405/ using document root <none> and password file <none>
I20260812 11:29:01.157579 14202 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:01.157734 14202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:01.158062 14202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:01.161077 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "c2ca971fe5ae49279296b7553e92024b"
format_stamp: "Formatted at 2026-08-12 11:29:01 on dist-test-slave-h6n0"
server_key: "00aa4b719a74ad7c1a78e8ca3a1ccec4"
server_key_iv: "c28ddd67ce8aadcdd4a8ad1b3cb1ff36"
server_key_version: "encryptionkey@0"
I20260812 11:29:01.161840 14202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "c2ca971fe5ae49279296b7553e92024b"
format_stamp: "Formatted at 2026-08-12 11:29:01 on dist-test-slave-h6n0"
server_key: "00aa4b719a74ad7c1a78e8ca3a1ccec4"
server_key_iv: "c28ddd67ce8aadcdd4a8ad1b3cb1ff36"
server_key_version: "encryptionkey@0"
I20260812 11:29:01.166971 14202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:29:01.170207 14228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:01.171504 14202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:29:01.171679 14202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "c2ca971fe5ae49279296b7553e92024b"
format_stamp: "Formatted at 2026-08-12 11:29:01 on dist-test-slave-h6n0"
server_key: "00aa4b719a74ad7c1a78e8ca3a1ccec4"
server_key_iv: "c28ddd67ce8aadcdd4a8ad1b3cb1ff36"
server_key_version: "encryptionkey@0"
I20260812 11:29:01.171875 14202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:01.187707 14202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:01.192186 14202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:01.192515 14202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:01.193504 14202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:29:01.194918 14202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:29:01.195019 14202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:01.195168 14202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:29:01.195245 14202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:01.231966 14202 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:46559
I20260812 11:29:01.232005 14424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:46559 every 8 connection(s)
I20260812 11:29:01.233253 14202 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:29:01.234287 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14202
I20260812 11:29:01.234486 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:29:01.235409 12731 external_mini_cluster.cc:1428] Setting key 2a80615bb05e87563052c2e01036e4ee
Aug 12 11:29:01 dist-test-slave-h6n0 krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534141, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.194@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:01.259436 14429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.236354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:50441 (local address 127.12.110.254:37531)
0812 11:29:01.242987 (+ 6633us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:01.242995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:01.243052 (+ 57us) server_negotiation.cc:407] Connection header received
0812 11:29:01.243120 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:01.243127 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:01.243218 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:01.243337 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:01.245176 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.246332 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.247997 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.248788 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.248929 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.253409 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:01.253417 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:01.253430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:01.253478 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:01.256257 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:01.256995 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:01.257005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:01.257007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:01.257141 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:01.257600 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:01.257609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:01.257611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:01.257895 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:01.258058 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:01.258833 (+ 775us) server_negotiation.cc:299] Negotiation successful
0812 11:29:01.259061 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6482,"thread_start_us":6355,"threads_started":1}
I20260812 11:29:01.260447 14432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.236753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37531 (local address 127.12.110.194:50441)
0812 11:29:01.238389 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:01.238455 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:01.242750 (+ 4295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:01.243695 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:01.243722 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:01.244404 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:01.244987 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.245006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.246508 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.246512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.247845 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.247859 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.249031 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.249036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.249236 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.249994 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:01.250026 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:01.253151 (+ 3125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:01.256421 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:01.256436 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:01.256449 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:01.256860 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:01.257261 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:01.257266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:01.257271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:01.257472 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:01.258032 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:01.258040 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:01.258393 (+ 353us) client_negotiation.cc:769] Sending connection context
0812 11:29:01.258715 (+ 322us) client_negotiation.cc:240] Negotiation successful
0812 11:29:01.259239 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":899,"thread_start_us":591,"threads_started":1}
I20260812 11:29:01.262055 14425 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37531
I20260812 11:29:01.262365 14425 heartbeater.cc:464] Registering TS with master...
I20260812 11:29:01.263278 14425 heartbeater.cc:511] Master 127.12.110.254:37531 requested a full tablet report, sending...
I20260812 11:29:01.264832 13859 ts_manager.cc:194] Registered new tserver with Master: c2ca971fe5ae49279296b7553e92024b (127.12.110.194:46559)
I20260812 11:29:01.266155 13859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.194@KRBTEST.COM'} at 127.12.110.194:50441
WARNING: no policy specified for kudu/127.12.110.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
Entry for principal kudu/127.12.110.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab.
I20260812 11:29:01.309674 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37531
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:37497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:01.656477 14435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:01.656760 14435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:01.656854 14435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:01.664049 14435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:01.664160 14435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:01.664276 14435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:29:01.674208 14435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:37497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.195.keytab
--principal=kudu/127.12.110.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:37531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:01.675616 14435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:01.676986 14435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:01.690824 14447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:01.690866 14448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:01.690928 14451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:01.691656 14435 server_base.cc:1034] running on GCE node
I20260812 11:29:01.692293 14435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:01.693293 14435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:01.694586 14435 hybrid_clock.cc:648] HybridClock initialized: now 1786534141694502 us; error 57 us; skew 500 ppm
I20260812 11:29:01.694924 14435 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:01 dist-test-slave-h6n0 krb5kdc[13768](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534141, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:01.698168 14435 init.cc:377] Logged in from keytab as kudu/127.12.110.195@KRBTEST.COM (short username kudu)
I20260812 11:29:01.699684 14435 webserver.cc:460] Webserver started at http://127.12.110.195:38647/ using document root <none> and password file <none>
I20260812 11:29:01.700331 14435 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:01.700434 14435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:01.700673 14435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:01.703552 14435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "79825da229d24e5faaa01483d2e1a5da"
format_stamp: "Formatted at 2026-08-12 11:29:01 on dist-test-slave-h6n0"
server_key: "d449d8d92bf1079d75fc223d10c46913"
server_key_iv: "9e8121471352f273c0f23aca2e8b25de"
server_key_version: "encryptionkey@0"
I20260812 11:29:01.704258 14435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "79825da229d24e5faaa01483d2e1a5da"
format_stamp: "Formatted at 2026-08-12 11:29:01 on dist-test-slave-h6n0"
server_key: "d449d8d92bf1079d75fc223d10c46913"
server_key_iv: "9e8121471352f273c0f23aca2e8b25de"
server_key_version: "encryptionkey@0"
I20260812 11:29:01.705708 14193 heartbeater.cc:503] Master 127.12.110.254:37531 was elected leader, sending a full tablet report...
I20260812 11:29:01.710280 14435 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 11:29:01.714335 14463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:01.716008 14435 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:29:01.716218 14435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "79825da229d24e5faaa01483d2e1a5da"
format_stamp: "Formatted at 2026-08-12 11:29:01 on dist-test-slave-h6n0"
server_key: "d449d8d92bf1079d75fc223d10c46913"
server_key_iv: "9e8121471352f273c0f23aca2e8b25de"
server_key_version: "encryptionkey@0"
I20260812 11:29:01.716423 14435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:01.757845 14435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:01.762370 14435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:01.762715 14435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:01.763746 14435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:29:01.765115 14435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:29:01.765193 14435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:01.765329 14435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:29:01.765383 14435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:01.800855 14435 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:44061
I20260812 11:29:01.800896 14651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:44061 every 8 connection(s)
I20260812 11:29:01.802295 14435 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:29:01.804257 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14435
I20260812 11:29:01.804484 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:29:01.805104 12731 external_mini_cluster.cc:1428] Setting key fe63f2f301db2db75fd608173aee4339
Aug 12 11:29:01 dist-test-slave-h6n0 krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534141, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.195@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:01.824851 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.805370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:44231 (local address 127.12.110.254:37531)
0812 11:29:01.805937 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:01.805943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:01.806933 (+ 990us) server_negotiation.cc:407] Connection header received
0812 11:29:01.809060 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:01.809068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:01.809306 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:01.809478 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:01.811415 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.812182 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.813663 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.814073 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.814167 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.818667 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:01.818672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:01.818676 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:01.818707 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:01.821942 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:01.822711 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:01.822715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:01.822717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:01.822849 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:01.823261 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:01.823264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:01.823266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:01.823561 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:01.823684 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:01.824383 (+ 699us) server_negotiation.cc:299] Negotiation successful
0812 11:29:01.824555 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":259,"threads_started":1}
I20260812 11:29:01.825721 14658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.805876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:37531 (local address 127.12.110.195:44231)
0812 11:29:01.806767 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:01.806812 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:01.808621 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:01.809700 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:01.809708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:01.810645 (+ 937us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:01.811242 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.811271 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.812335 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.812339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.813524 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.813535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.814267 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.814272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.814449 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.815266 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:01.815301 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:01.818408 (+ 3107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:01.822080 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:01.822086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:01.822097 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:01.822609 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:01.822955 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:01.822958 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:01.822961 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:01.823152 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:01.823698 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:01.823704 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:01.824011 (+ 307us) client_negotiation.cc:769] Sending connection context
0812 11:29:01.824229 (+ 218us) client_negotiation.cc:240] Negotiation successful
0812 11:29:01.824587 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":397,"threads_started":1}
I20260812 11:29:01.827198 14654 heartbeater.cc:347] Connected to a master server at 127.12.110.254:37531
I20260812 11:29:01.827545 14654 heartbeater.cc:464] Registering TS with master...
I20260812 11:29:01.828325 14654 heartbeater.cc:511] Master 127.12.110.254:37531 requested a full tablet report, sending...
I20260812 11:29:01.829876 13859 ts_manager.cc:194] Registered new tserver with Master: 79825da229d24e5faaa01483d2e1a5da (127.12.110.195:44061)
I20260812 11:29:01.830801 13859 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.12.110.195@KRBTEST.COM'} at 127.12.110.195:44231
I20260812 11:29:01.833086 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:29:01.851150 14656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.837158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.12.110.254:37531)
0812 11:29:01.837323 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:01.837328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:01.837843 (+ 515us) server_negotiation.cc:407] Connection header received
0812 11:29:01.838044 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:01.838050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:01.838136 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:01.838238 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:01.839692 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.840423 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.841744 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.842150 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.842290 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.844991 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:01.844996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:01.844999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:01.845040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:01.848365 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:01.849043 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:01.849049 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:01.849053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:01.849178 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:01.849558 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:01.849563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:01.849566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:01.849927 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:01.850228 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:01.850586 (+ 358us) server_negotiation.cc:299] Negotiation successful
0812 11:29:01.850825 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:29:01.855263 13859 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:29:01.862073 13859 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:29:01.899807 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.876193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.12.110.193:35579)
0812 11:29:01.891427 (+ 15234us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:01.891438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:01.891465 (+ 27us) server_negotiation.cc:407] Connection header received
0812 11:29:01.891557 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:01.891563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:01.891781 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:01.891982 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:01.892938 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.894616 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.897333 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.898503 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.898812 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.898979 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:01.899300 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 11:29:01.899470 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14973,"thread_start_us":14704,"threads_started":1}
I20260812 11:29:01.899967 14673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.875802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:35579 (local address 127.0.0.1:50498)
0812 11:29:01.876659 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:01.876705 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:01.876980 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:01.892038 (+ 15058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:01.892050 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:01.892102 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:01.892723 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.892736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.894773 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.894779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.897179 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.897192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.898777 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.898784 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.899011 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.899108 (+ 97us) client_negotiation.cc:769] Sending connection context
0812 11:29:01.899242 (+ 134us) client_negotiation.cc:240] Negotiation successful
0812 11:29:01.899447 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":48,"thread_start_us":732,"threads_started":1}
I20260812 11:29:01.899883 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.876872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60220 (local address 127.12.110.195:44061)
0812 11:29:01.889456 (+ 12584us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:01.889469 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:01.890743 (+ 1274us) server_negotiation.cc:407] Connection header received
0812 11:29:01.890991 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:01.891002 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:01.891302 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:01.891515 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:01.892681 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.894107 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.897098 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.898226 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.898380 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.898596 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:01.899313 (+ 717us) server_negotiation.cc:299] Negotiation successful
0812 11:29:01.899480 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12336,"thread_start_us":12613,"threads_started":1}
I20260812 11:29:01.900961 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.877933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.12.110.194:46559)
0812 11:29:01.890152 (+ 12219us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:01.890157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:01.892814 (+ 2657us) server_negotiation.cc:407] Connection header received
0812 11:29:01.893133 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:01.893143 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:01.893333 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:01.893481 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:01.894275 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.895426 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.898104 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.899607 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.899767 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.899919 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:01.900476 (+ 557us) server_negotiation.cc:299] Negotiation successful
0812 11:29:01.900630 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12059,"thread_start_us":11927,"threads_started":1}
I20260812 11:29:01.901038 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.876126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:44061 (local address 127.0.0.1:60220)
0812 11:29:01.890604 (+ 14478us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:01.890624 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:01.890803 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:01.891840 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:01.891848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:01.891882 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:01.892394 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.892414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.894244 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.894251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.896908 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.896923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.898841 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.898850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.899040 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.899124 (+ 84us) client_negotiation.cc:769] Sending connection context
0812 11:29:01.900476 (+ 1352us) client_negotiation.cc:240] Negotiation successful
0812 11:29:01.900610 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14317,"mutex_wait_us":127,"thread_start_us":16207,"threads_started":1}
I20260812 11:29:01.901965 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:01.877606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:46559 (local address 127.0.0.1:35952)
0812 11:29:01.892695 (+ 15089us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:01.892719 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:01.892923 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:01.893713 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:01.893718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:01.893743 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:01.894085 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.894096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.895601 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.895605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.897920 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:01.897935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:01.900059 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:01.900065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:01.900242 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:01.900277 (+ 35us) client_negotiation.cc:769] Sending connection context
0812 11:29:01.901484 (+ 1207us) client_negotiation.cc:240] Negotiation successful
0812 11:29:01.901626 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14933,"mutex_wait_us":60,"thread_start_us":12705,"threads_started":1}
I20260812 11:29:01.904199 14540 tablet_service.cc:1467] Processing CreateTablet for tablet dd29f325169f45b096039ff10bd97de8 (DEFAULT_TABLE table=test-table [id=32f0061208bc42ee9105d295f4944c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:29:01.904284 14077 tablet_service.cc:1467] Processing CreateTablet for tablet dd29f325169f45b096039ff10bd97de8 (DEFAULT_TABLE table=test-table [id=32f0061208bc42ee9105d295f4944c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:29:01.905576 14540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd29f325169f45b096039ff10bd97de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:01.905651 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd29f325169f45b096039ff10bd97de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:01.905401 14321 tablet_service.cc:1467] Processing CreateTablet for tablet dd29f325169f45b096039ff10bd97de8 (DEFAULT_TABLE table=test-table [id=32f0061208bc42ee9105d295f4944c3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:29:01.906641 14321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd29f325169f45b096039ff10bd97de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:01.918143 14683 tablet_bootstrap.cc:492] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Bootstrap starting.
I20260812 11:29:01.921712 14683 tablet_bootstrap.cc:654] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:01.922065 14685 tablet_bootstrap.cc:492] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Bootstrap starting.
I20260812 11:29:01.922892 14689 tablet_bootstrap.cc:492] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Bootstrap starting.
I20260812 11:29:01.923532 14683 log.cc:824] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:01.927786 14685 tablet_bootstrap.cc:654] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:01.929117 14689 tablet_bootstrap.cc:654] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:01.929342 14685 log.cc:824] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:01.930192 14689 log.cc:824] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:01.932770 14685 tablet_bootstrap.cc:492] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: No bootstrap required, opened a new log
I20260812 11:29:01.933115 14685 ts_tablet_manager.cc:1397] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260812 11:29:01.933363 14683 tablet_bootstrap.cc:492] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: No bootstrap required, opened a new log
I20260812 11:29:01.933648 14689 tablet_bootstrap.cc:492] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: No bootstrap required, opened a new log
I20260812 11:29:01.933630 14683 ts_tablet_manager.cc:1397] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260812 11:29:01.933888 14689 ts_tablet_manager.cc:1397] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260812 11:29:01.940763 14685 raft_consensus.cc:348] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:01.941188 14685 raft_consensus.cc:374] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:01.941294 14685 raft_consensus.cc:729] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2ca971fe5ae49279296b7553e92024b, State: Initialized, Role: FOLLOWER
I20260812 11:29:01.941913 14685 consensus_queue.cc:260] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:01.944569 14425 heartbeater.cc:503] Master 127.12.110.254:37531 was elected leader, sending a full tablet report...
I20260812 11:29:01.944491 14683 raft_consensus.cc:348] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:01.944491 14689 raft_consensus.cc:348] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:01.944900 14683 raft_consensus.cc:374] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:01.944891 14689 raft_consensus.cc:374] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:01.944988 14683 raft_consensus.cc:729] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79825da229d24e5faaa01483d2e1a5da, State: Initialized, Role: FOLLOWER
I20260812 11:29:01.944994 14689 raft_consensus.cc:729] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cae887b58cad4f408c93fda9148f0a60, State: Initialized, Role: FOLLOWER
I20260812 11:29:01.945209 14685 ts_tablet_manager.cc:1428] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260812 11:29:01.945549 14683 consensus_queue.cc:260] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:01.945611 14689 consensus_queue.cc:260] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:01.946630 14654 heartbeater.cc:503] Master 127.12.110.254:37531 was elected leader, sending a full tablet report...
I20260812 11:29:01.947499 14683 ts_tablet_manager.cc:1428] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260812 11:29:01.949628 14689 ts_tablet_manager.cc:1428] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
W20260812 11:29:01.985428 14428 tablet.cc:2369] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:29:02.038626 14693 raft_consensus.cc:482] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:29:02.039014 14693 raft_consensus.cc:504] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:02.040745 14693 leader_election.cc:288] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cae887b58cad4f408c93fda9148f0a60 (127.12.110.193:35579), 79825da229d24e5faaa01483d2e1a5da (127.12.110.195:44061)
I20260812 11:29:02.048362 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.041332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:60421 (local address 127.12.110.195:44061)
0812 11:29:02.041581 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:02.041589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:02.042300 (+ 711us) server_negotiation.cc:407] Connection header received
0812 11:29:02.042453 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:02.042458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:02.042621 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:02.042719 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:02.043385 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.044153 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.046365 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.047325 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.047493 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.047584 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:02.047928 (+ 344us) server_negotiation.cc:299] Negotiation successful
0812 11:29:02.048090 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:29:02.048370 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.041319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:34689 (local address 127.12.110.193:35579)
0812 11:29:02.041576 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:02.041586 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:02.042482 (+ 896us) server_negotiation.cc:407] Connection header received
0812 11:29:02.042666 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:02.042670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:02.042762 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:02.042862 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:02.043538 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.044257 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.046317 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.047339 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.047531 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.047607 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:02.047989 (+ 382us) server_negotiation.cc:299] Negotiation successful
0812 11:29:02.048125 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 11:29:02.048511 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.041208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:44061 (local address 127.12.110.194:60421)
0812 11:29:02.042180 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:02.042196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:02.042317 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:02.042753 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:02.042757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:02.042780 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:02.043219 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:02.043231 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.044296 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.044304 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:02.046198 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:02.046210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.047574 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.047580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:02.047749 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.047792 (+ 43us) client_negotiation.cc:769] Sending connection context
0812 11:29:02.047866 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 11:29:02.048012 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":85,"thread_start_us":671,"threads_started":1}
I20260812 11:29:02.048542 14698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.041332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:35579 (local address 127.12.110.194:34689)
0812 11:29:02.042355 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:02.042374 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:02.042520 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:02.042905 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:02.042908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:02.042941 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:02.043370 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:02.043381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.044401 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.044408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:02.046140 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:02.046154 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.047618 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.047624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:02.047802 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.047825 (+ 23us) client_negotiation.cc:769] Sending connection context
0812 11:29:02.047944 (+ 119us) client_negotiation.cc:240] Negotiation successful
0812 11:29:02.048085 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":885,"mutex_wait_us":29,"thread_start_us":463,"threads_started":1}
I20260812 11:29:02.049798 14112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29f325169f45b096039ff10bd97de8" candidate_uuid: "c2ca971fe5ae49279296b7553e92024b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae887b58cad4f408c93fda9148f0a60" is_pre_election: true
I20260812 11:29:02.049875 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29f325169f45b096039ff10bd97de8" candidate_uuid: "c2ca971fe5ae49279296b7553e92024b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79825da229d24e5faaa01483d2e1a5da" is_pre_election: true
I20260812 11:29:02.050427 14112 raft_consensus.cc:2454] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ca971fe5ae49279296b7553e92024b in term 0.
I20260812 11:29:02.050427 14576 raft_consensus.cc:2454] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2ca971fe5ae49279296b7553e92024b in term 0.
I20260812 11:29:02.051170 14235 leader_election.cc:302] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ca971fe5ae49279296b7553e92024b, cae887b58cad4f408c93fda9148f0a60; no voters:
I20260812 11:29:02.051585 14693 raft_consensus.cc:2781] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:29:02.051733 14693 raft_consensus.cc:482] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:29:02.051846 14693 raft_consensus.cc:3037] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:02.054061 14693 raft_consensus.cc:504] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
W20260812 11:29:02.055123 14655 tablet.cc:2369] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:29:02.055212 14112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29f325169f45b096039ff10bd97de8" candidate_uuid: "c2ca971fe5ae49279296b7553e92024b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cae887b58cad4f408c93fda9148f0a60"
I20260812 11:29:02.055276 14576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd29f325169f45b096039ff10bd97de8" candidate_uuid: "c2ca971fe5ae49279296b7553e92024b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79825da229d24e5faaa01483d2e1a5da"
I20260812 11:29:02.055429 14112 raft_consensus.cc:3037] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:02.055514 14576 raft_consensus.cc:3037] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:02.054790 14693 leader_election.cc:288] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [CANDIDATE]: Term 1 election: Requested vote from peers cae887b58cad4f408c93fda9148f0a60 (127.12.110.193:35579), 79825da229d24e5faaa01483d2e1a5da (127.12.110.195:44061)
I20260812 11:29:02.057799 14112 raft_consensus.cc:2454] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ca971fe5ae49279296b7553e92024b in term 1.
I20260812 11:29:02.057829 14576 raft_consensus.cc:2454] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2ca971fe5ae49279296b7553e92024b in term 1.
I20260812 11:29:02.058209 14235 leader_election.cc:302] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2ca971fe5ae49279296b7553e92024b, cae887b58cad4f408c93fda9148f0a60; no voters:
I20260812 11:29:02.058492 14693 raft_consensus.cc:2781] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:02.058996 14693 raft_consensus.cc:686] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 1 LEADER]: Becoming Leader. State: Replica: c2ca971fe5ae49279296b7553e92024b, State: Running, Role: LEADER
I20260812 11:29:02.059473 14693 consensus_queue.cc:237] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } }
I20260812 11:29:02.064132 13857 catalog_manager.cc:5495] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b reported cstate change: term changed from 0 to 1, leader changed from <none> to c2ca971fe5ae49279296b7553e92024b (127.12.110.194). New cstate: current_term: 1 leader_uuid: "c2ca971fe5ae49279296b7553e92024b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2ca971fe5ae49279296b7553e92024b" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 46559 } health_report { overall_health: HEALTHY } } }
I20260812 11:29:02.095942 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.090735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.12.110.194:46559)
0812 11:29:02.090918 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:02.090923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:02.091009 (+ 86us) server_negotiation.cc:407] Connection header received
0812 11:29:02.091205 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:02.091208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:02.091285 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:02.091409 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:02.091718 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.092419 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.093790 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.094334 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.094450 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.094915 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:02.095408 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:02.095548 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:02.095660 (+ 112us) server_negotiation.cc:299] Negotiation successful
0812 11:29:02.095727 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:29:02.104003 14112 raft_consensus.cc:1261] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 1 FOLLOWER]: Refusing update from remote peer c2ca971fe5ae49279296b7553e92024b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:02.104004 14576 raft_consensus.cc:1261] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 1 FOLLOWER]: Refusing update from remote peer c2ca971fe5ae49279296b7553e92024b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:02.105083 14700 consensus_queue.cc:1035] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cae887b58cad4f408c93fda9148f0a60" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 35579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:29:02.105480 14693 consensus_queue.cc:1035] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79825da229d24e5faaa01483d2e1a5da" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 44061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:29:02.129588 13859 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:29:02.129932 13859 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:29:02.133738 13859 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c685e70067c4e3a9e70e0fab0104443: Sending DeleteTablet for 3 replicas of tablet dd29f325169f45b096039ff10bd97de8
I20260812 11:29:02.134902 14540 tablet_service.cc:1514] Processing DeleteTablet for tablet dd29f325169f45b096039ff10bd97de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:29:02 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:60220
I20260812 11:29:02.134969 14077 tablet_service.cc:1514] Processing DeleteTablet for tablet dd29f325169f45b096039ff10bd97de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:29:02 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:50498
I20260812 11:29:02.135152 14321 tablet_service.cc:1514] Processing DeleteTablet for tablet dd29f325169f45b096039ff10bd97de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:29:02 UTC) from {username='kudu', principal='kudu/127.12.110.254@KRBTEST.COM'} at 127.0.0.1:35952
I20260812 11:29:02.136444 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13966
I20260812 11:29:02.156976 14720 tablet_replica.cc:330] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: stopping tablet replica
I20260812 11:29:02.157435 14720 raft_consensus.cc:2229] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:29:02.158000 14720 raft_consensus.cc:2258] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:29:02.159055 14723 tablet_replica.cc:330] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: stopping tablet replica
I20260812 11:29:02.159337 14722 tablet_replica.cc:330] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: stopping tablet replica
I20260812 11:29:02.159509 14723 raft_consensus.cc:2229] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:29:02.159780 14722 raft_consensus.cc:2229] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:29:02.160254 14722 raft_consensus.cc:2258] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:29:02.160745 14723 raft_consensus.cc:2258] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:29:02.162745 14723 ts_tablet_manager.cc:1905] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:29:02.162727 14722 ts_tablet_manager.cc:1905] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:29:02.162729 14720 ts_tablet_manager.cc:1905] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 12 11:29:02 dist-test-slave-h6n0 krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.193@KRBTEST.COM
I20260812 11:29:02.170096 14722 ts_tablet_manager.cc:1918] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:29:02.170210 14723 ts_tablet_manager.cc:1918] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:29:02.170411 14722 log.cc:1196] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-0/wal/wals/dd29f325169f45b096039ff10bd97de8
I20260812 11:29:02.170450 14723 log.cc:1196] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/dd29f325169f45b096039ff10bd97de8
I20260812 11:29:02.171096 14723 ts_tablet_manager.cc:1939] T dd29f325169f45b096039ff10bd97de8 P c2ca971fe5ae49279296b7553e92024b: Deleting consensus metadata
I20260812 11:29:02.171170 14722 ts_tablet_manager.cc:1939] T dd29f325169f45b096039ff10bd97de8 P cae887b58cad4f408c93fda9148f0a60: Deleting consensus metadata
I20260812 11:29:02.173009 14720 ts_tablet_manager.cc:1918] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:29:02.173132 13835 catalog_manager.cc:4841] TS c2ca971fe5ae49279296b7553e92024b (127.12.110.194:46559): tablet dd29f325169f45b096039ff10bd97de8 (table test-table [id=32f0061208bc42ee9105d295f4944c3e]) successfully deleted
I20260812 11:29:02.173487 13835 catalog_manager.cc:4841] TS cae887b58cad4f408c93fda9148f0a60 (127.12.110.193:35579): tablet dd29f325169f45b096039ff10bd97de8 (table test-table [id=32f0061208bc42ee9105d295f4944c3e]) successfully deleted
I20260812 11:29:02.174350 14720 log.cc:1196] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/dd29f325169f45b096039ff10bd97de8
I20260812 11:29:02.174952 14720 ts_tablet_manager.cc:1939] T dd29f325169f45b096039ff10bd97de8 P 79825da229d24e5faaa01483d2e1a5da: Deleting consensus metadata
I20260812 11:29:02.176954 13828 catalog_manager.cc:4841] TS 79825da229d24e5faaa01483d2e1a5da (127.12.110.195:44061): tablet dd29f325169f45b096039ff10bd97de8 (table test-table [id=32f0061208bc42ee9105d295f4944c3e]) successfully deleted
I20260812 11:29:02.181736 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.137810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.12.110.193:35579)
0812 11:29:02.138051 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:02.138060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:02.148781 (+ 10721us) server_negotiation.cc:407] Connection header received
0812 11:29:02.149749 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:02.149760 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:02.149880 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:02.149974 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:02.152510 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.154058 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.158444 (+ 4386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.159271 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.159388 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.174408 (+ 15020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:02.174424 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:02.174442 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:02.174502 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:02.178648 (+ 4146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:02.179422 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:02.179434 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:02.179437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:02.179616 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:02.180020 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:02.180026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:02.180029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:02.180542 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:02.180900 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:02.181111 (+ 211us) server_negotiation.cc:299] Negotiation successful
0812 11:29:02.181345 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260812 11:29:02.182554 14177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:02.294024 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13966
I20260812 11:29:02.327464 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14202
Aug 12 11:29:02 dist-test-slave-h6n0 krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.194@KRBTEST.COM
I20260812 11:29:02.348367 14680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.328903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.12.110.194:46559)
0812 11:29:02.329173 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:02.329180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:02.329205 (+ 25us) server_negotiation.cc:407] Connection header received
0812 11:29:02.329363 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:02.329369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:02.329455 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:02.329567 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:02.331234 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.332510 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.334153 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.334858 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.335002 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.341976 (+ 6974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:02.341988 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:02.342003 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:02.342054 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:02.345319 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:02.346064 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:02.346077 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:02.346080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:02.346224 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:02.346773 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:02.346780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:02.346783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:02.347138 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:02.347478 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:02.347760 (+ 282us) server_negotiation.cc:299] Negotiation successful
0812 11:29:02.347987 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 11:29:02.349035 14420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:02.463053 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14202
I20260812 11:29:02.496430 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14435
Aug 12 11:29:02 dist-test-slave-h6n0 krb5kdc[13768](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.195@KRBTEST.COM
I20260812 11:29:02.512167 14677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:02.497231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60226 (local address 127.12.110.195:44061)
0812 11:29:02.497414 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:02.497420 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:02.497520 (+ 100us) server_negotiation.cc:407] Connection header received
0812 11:29:02.497733 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:02.497745 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:02.497819 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:02.497952 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:02.499404 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.500150 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.501480 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:02.501860 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:02.501997 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:02.506747 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:02.506759 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:02.506775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:02.506807 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:02.509356 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:02.510025 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:02.510032 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:02.510034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:02.510181 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:02.510577 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:02.510582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:02.510585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:02.510966 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:02.511292 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:02.511587 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 11:29:02.511809 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:29:02.512907 14643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:02.600186 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14435
I20260812 11:29:02.625721 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 13799
I20260812 11:29:02.627048 13917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:02.738040 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 13799
2026-08-12T11:29:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4938 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-12T11:29:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:02Z Disabled control of system clock
I20260812 11:29:02.840883 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41681
--webserver_interface=127.12.110.254
--webserver_port=0
--builtin_ntp_servers=127.12.110.212:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:41681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:29:03.185300 14735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:03.185611 14735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:03.185726 14735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:03.193068 14735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:03.193173 14735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:03.193257 14735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:03.193367 14735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:03.193436 14735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:03.202991 14735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:41681
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:41681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:03.204423 14735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:03.206251 14735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:03.219480 14743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:03.219599 14740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:03.219529 14748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:03.220777 14735 server_base.cc:1034] running on GCE node
I20260812 11:29:03.221586 14735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:03.223044 14735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:03.224244 14735 hybrid_clock.cc:648] HybridClock initialized: now 1786534143224189 us; error 47 us; skew 500 ppm
I20260812 11:29:03.224596 14735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:29:03.226912 14735 webserver.cc:460] Webserver started at http://127.12.110.254:44143/ using document root <none> and password file <none>
I20260812 11:29:03.227605 14735 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:03.227696 14735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:03.228092 14735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:03.231063 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "e2edfb13f3ff4832936f2674975a3dbb"
format_stamp: "Formatted at 2026-08-12 11:29:03 on dist-test-slave-h6n0"
server_key: "9428c18c64f3a2ce1c021173b36a86d0"
server_key_iv: "926ae14839e599599577122274ac6d35"
server_key_version: "encryptionkey@0"
I20260812 11:29:03.231894 14735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "e2edfb13f3ff4832936f2674975a3dbb"
format_stamp: "Formatted at 2026-08-12 11:29:03 on dist-test-slave-h6n0"
server_key: "9428c18c64f3a2ce1c021173b36a86d0"
server_key_iv: "926ae14839e599599577122274ac6d35"
server_key_version: "encryptionkey@0"
I20260812 11:29:03.237062 14735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260812 11:29:03.240754 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:03.242041 14735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:29:03.242261 14735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "e2edfb13f3ff4832936f2674975a3dbb"
format_stamp: "Formatted at 2026-08-12 11:29:03 on dist-test-slave-h6n0"
server_key: "9428c18c64f3a2ce1c021173b36a86d0"
server_key_iv: "926ae14839e599599577122274ac6d35"
server_key_version: "encryptionkey@0"
I20260812 11:29:03.242534 14735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:03.275435 14735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:03.276342 14735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:03.276669 14735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:03.298475 14735 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:41681
I20260812 11:29:03.298503 14851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:41681 every 8 connection(s)
I20260812 11:29:03.299916 14735 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:03.303810 14853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:03.309638 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14735
I20260812 11:29:03.309856 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:03.310320 12731 external_mini_cluster.cc:1428] Setting key be02eba64ed988e436283b599940acfa
I20260812 11:29:03.316427 14853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb: Bootstrap starting.
I20260812 11:29:03.320495 14853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:03.321578 14853 log.cc:824] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:03.325296 14853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb: No bootstrap required, opened a new log
I20260812 11:29:03.327589 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:03.312861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32878 (local address 127.12.110.254:41681)
0812 11:29:03.314214 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:03.314229 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:03.314278 (+ 49us) server_negotiation.cc:407] Connection header received
0812 11:29:03.316727 (+ 2449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:03.316757 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:03.317314 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:03.317753 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:03.318329 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:03.320284 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:03.322225 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:03.323225 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:03.323434 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:03.324641 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:03.324658 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:03.324676 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:03.325441 (+ 765us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:03.325546 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:03.325592 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:03.325710 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:03.325956 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 11:29:03.326287 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":462,"threads_started":1}
I20260812 11:29:03.334812 14853 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2edfb13f3ff4832936f2674975a3dbb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41681 } }
I20260812 11:29:03.335207 14853 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:03.335331 14853 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2edfb13f3ff4832936f2674975a3dbb, State: Initialized, Role: FOLLOWER
I20260812 11:29:03.336019 14853 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2edfb13f3ff4832936f2674975a3dbb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41681 } }
I20260812 11:29:03.336431 14853 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:03.336606 14853 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:03.336798 14853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:03.339221 14853 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2edfb13f3ff4832936f2674975a3dbb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41681 } }
I20260812 11:29:03.339709 14853 leader_election.cc:302] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2edfb13f3ff4832936f2674975a3dbb; no voters:
I20260812 11:29:03.340344 14853 leader_election.cc:288] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:03.340422 14863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:03.341672 14863 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [term 1 LEADER]: Becoming Leader. State: Replica: e2edfb13f3ff4832936f2674975a3dbb, State: Running, Role: LEADER
I20260812 11:29:03.342199 14863 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2edfb13f3ff4832936f2674975a3dbb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41681 } }
I20260812 11:29:03.342958 14853 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:03.345192 14867 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2edfb13f3ff4832936f2674975a3dbb. Latest consensus state: current_term: 1 leader_uuid: "e2edfb13f3ff4832936f2674975a3dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2edfb13f3ff4832936f2674975a3dbb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41681 } } }
I20260812 11:29:03.345319 14865 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2edfb13f3ff4832936f2674975a3dbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2edfb13f3ff4832936f2674975a3dbb" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 41681 } } }
I20260812 11:29:03.345739 14865 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:03.345620 14867 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:03.349622 14873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:03.355680 14873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:03.365672 14873 catalog_manager.cc:1347] Generated new cluster ID: 92ea244bc092488ba9fcd0b1a0c72d75
I20260812 11:29:03.365819 14873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:03.374145 14873 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:03.375474 14873 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:03.387125 14873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb: Generated new TSK 0
I20260812 11:29:03.388096 14873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:03.400241 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.193:0
--local_ip_for_outbound_sockets=127.12.110.193
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41681
--builtin_ntp_servers=127.12.110.212:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260812 11:29:03.770198 14890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:03.770553 14890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:03.770722 14890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:03.778504 14890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:03.778674 14890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:03.778858 14890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.193
I20260812 11:29:03.787953 14890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.12.110.193
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.193
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:03.789227 14890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:03.791116 14890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:03.804785 14901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:03.805328 14898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:03.805411 14899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:03.806715 14890 server_base.cc:1034] running on GCE node
I20260812 11:29:03.807396 14890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:03.808414 14890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:03.809625 14890 hybrid_clock.cc:648] HybridClock initialized: now 1786534143809589 us; error 38 us; skew 500 ppm
I20260812 11:29:03.809957 14890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:29:03.812417 14890 webserver.cc:460] Webserver started at http://127.12.110.193:38101/ using document root <none> and password file <none>
I20260812 11:29:03.813115 14890 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:03.813232 14890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:03.813568 14890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:03.816573 14890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data/instance:
uuid: "33f71f1d8d284065a5b103b595a8021f"
format_stamp: "Formatted at 2026-08-12 11:29:03 on dist-test-slave-h6n0"
server_key: "36c6fe1bcf650a3813c8fe55fdbe62f7"
server_key_iv: "5003d78640c013197e12f490e87ce19f"
server_key_version: "encryptionkey@0"
I20260812 11:29:03.817314 14890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance:
uuid: "33f71f1d8d284065a5b103b595a8021f"
format_stamp: "Formatted at 2026-08-12 11:29:03 on dist-test-slave-h6n0"
server_key: "36c6fe1bcf650a3813c8fe55fdbe62f7"
server_key_iv: "5003d78640c013197e12f490e87ce19f"
server_key_version: "encryptionkey@0"
I20260812 11:29:03.822263 14890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260812 11:29:03.825842 14907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:03.827272 14890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:29:03.827458 14890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/wal
uuid: "33f71f1d8d284065a5b103b595a8021f"
format_stamp: "Formatted at 2026-08-12 11:29:03 on dist-test-slave-h6n0"
server_key: "36c6fe1bcf650a3813c8fe55fdbe62f7"
server_key_iv: "5003d78640c013197e12f490e87ce19f"
server_key_version: "encryptionkey@0"
I20260812 11:29:03.827689 14890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:03.842655 14890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:03.843502 14890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:03.843817 14890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:03.844822 14890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:29:03.846334 14890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:29:03.846460 14890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:03.846613 14890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:29:03.846714 14890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:03.882752 14890 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.193:36711
I20260812 11:29:03.882792 15102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.193:36711 every 8 connection(s)
I20260812 11:29:03.884195 14890 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/data/info.pb
I20260812 11:29:03.885201 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 14890
I20260812 11:29:03.885429 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-0/wal/instance
I20260812 11:29:03.885951 12731 external_mini_cluster.cc:1428] Setting key 1cecd431e54f201239e2d47fd79448dd
I20260812 11:29:03.889235 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.194:0
--local_ip_for_outbound_sockets=127.12.110.194
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41681
--builtin_ntp_servers=127.12.110.212:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:29:03.898488 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:03.887002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.193:59891 (local address 127.12.110.254:41681)
0812 11:29:03.888175 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:03.888182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:03.889144 (+ 962us) server_negotiation.cc:407] Connection header received
0812 11:29:03.891092 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:03.891101 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:03.891216 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:03.891348 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:03.892347 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:03.893526 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:03.895143 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:03.895920 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:03.896083 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:03.897454 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:03.897463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:03.897471 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:03.897497 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:03.897556 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:03.897577 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:03.897689 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:03.897895 (+ 206us) server_negotiation.cc:299] Negotiation successful
0812 11:29:03.898011 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":909,"threads_started":1}
I20260812 11:29:03.899577 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:03.887813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41681 (local address 127.12.110.193:59891)
0812 11:29:03.888961 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:03.889018 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:03.890829 (+ 1811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:03.891560 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:03.891567 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:03.891643 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:03.892172 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:03.892188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:03.893704 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:03.893708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:03.894995 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:03.895008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:03.896147 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:03.896152 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:03.896336 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:03.897132 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:29:03.897135 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:03.897171 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:29:03.897241 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:03.897689 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:03.897697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:03.897702 (+ 5us) client_negotiation.cc:769] Sending connection context
0812 11:29:03.897839 (+ 137us) client_negotiation.cc:240] Negotiation successful
0812 11:29:03.898269 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":379,"threads_started":1}
I20260812 11:29:03.901068 15104 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41681
I20260812 11:29:03.901367 15104 heartbeater.cc:464] Registering TS with master...
I20260812 11:29:03.902093 15104 heartbeater.cc:511] Master 127.12.110.254:41681 requested a full tablet report, sending...
I20260812 11:29:03.904057 14791 ts_manager.cc:194] Registered new tserver with Master: 33f71f1d8d284065a5b103b595a8021f (127.12.110.193:36711)
I20260812 11:29:03.905436 14791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.193:59891
W20260812 11:29:04.239029 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:04.239302 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:04.239439 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:04.246876 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:04.246980 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:04.247160 15109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.194
I20260812 11:29:04.257182 15109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.12.110.194
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.194
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:04.258426 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:04.260181 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:04.273118 15118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:04.273365 15117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:04.273782 15121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:04.275081 15109 server_base.cc:1034] running on GCE node
I20260812 11:29:04.275844 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:04.277091 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:04.278333 15109 hybrid_clock.cc:648] HybridClock initialized: now 1786534144278260 us; error 50 us; skew 500 ppm
I20260812 11:29:04.278791 15109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:29:04.281778 15109 webserver.cc:460] Webserver started at http://127.12.110.194:34629/ using document root <none> and password file <none>
I20260812 11:29:04.282406 15109 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:04.282493 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:04.282857 15109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:04.285560 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data/instance:
uuid: "444f8aaf7128463ea9a2446d036c31df"
format_stamp: "Formatted at 2026-08-12 11:29:04 on dist-test-slave-h6n0"
server_key: "2be0ad87c26253a5981b52dea73594e8"
server_key_iv: "c22547330b1788cfc9a237773faf2c6a"
server_key_version: "encryptionkey@0"
I20260812 11:29:04.286273 15109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance:
uuid: "444f8aaf7128463ea9a2446d036c31df"
format_stamp: "Formatted at 2026-08-12 11:29:04 on dist-test-slave-h6n0"
server_key: "2be0ad87c26253a5981b52dea73594e8"
server_key_iv: "c22547330b1788cfc9a237773faf2c6a"
server_key_version: "encryptionkey@0"
I20260812 11:29:04.291150 15109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:29:04.294512 15128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:04.295810 15109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:29:04.295981 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal
uuid: "444f8aaf7128463ea9a2446d036c31df"
format_stamp: "Formatted at 2026-08-12 11:29:04 on dist-test-slave-h6n0"
server_key: "2be0ad87c26253a5981b52dea73594e8"
server_key_iv: "c22547330b1788cfc9a237773faf2c6a"
server_key_version: "encryptionkey@0"
I20260812 11:29:04.296190 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:04.334287 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:04.335321 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:04.335677 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:04.336747 15109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:29:04.338294 15109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:29:04.338425 15109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:04.338594 15109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:29:04.338690 15109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:04.373104 15109 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.194:35113
I20260812 11:29:04.373198 15311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.194:35113 every 8 connection(s)
I20260812 11:29:04.374472 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/data/info.pb
I20260812 11:29:04.384500 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15109
I20260812 11:29:04.384699 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal/instance
I20260812 11:29:04.385303 12731 external_mini_cluster.cc:1428] Setting key 01ca87ade848798fb23178f48d1fbec2
I20260812 11:29:04.386938 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.377145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:54585 (local address 127.12.110.254:41681)
0812 11:29:04.377375 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:04.377382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:04.378858 (+ 1476us) server_negotiation.cc:407] Connection header received
0812 11:29:04.380654 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:04.380659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:04.380726 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:04.380814 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:04.381788 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.382491 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.384013 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.384431 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.384527 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.386118 (+ 1591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:04.386126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:04.386132 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:04.386153 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:04.386194 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:04.386210 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:04.386306 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:04.386624 (+ 318us) server_negotiation.cc:299] Negotiation successful
0812 11:29:04.386709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260812 11:29:04.388034 15314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.377754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41681 (local address 127.12.110.194:54585)
0812 11:29:04.378692 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:04.378738 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:04.380396 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:04.381008 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:04.381015 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:04.381088 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:04.381637 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.381654 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.382682 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.382685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.383875 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.383892 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.384666 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.384674 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.384962 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.385778 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:29:04.385782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:04.385828 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:29:04.385927 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:04.386321 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:04.386330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:04.386337 (+ 7us) client_negotiation.cc:769] Sending connection context
0812 11:29:04.386579 (+ 242us) client_negotiation.cc:240] Negotiation successful
0812 11:29:04.386928 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":288,"threads_started":1}
I20260812 11:29:04.388411 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.12.110.195:0
--local_ip_for_outbound_sockets=127.12.110.195
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41681
--builtin_ntp_servers=127.12.110.212:36515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260812 11:29:04.389391 15312 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41681
I20260812 11:29:04.389691 15312 heartbeater.cc:464] Registering TS with master...
I20260812 11:29:04.390369 15312 heartbeater.cc:511] Master 127.12.110.254:41681 requested a full tablet report, sending...
I20260812 11:29:04.391780 14791 ts_manager.cc:194] Registered new tserver with Master: 444f8aaf7128463ea9a2446d036c31df (127.12.110.194:35113)
I20260812 11:29:04.392560 14791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.194:54585
W20260812 11:29:04.743358 15315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:04.743714 15315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:04.743880 15315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:04.751284 15315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:04.751379 15315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:04.751576 15315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.12.110.195
I20260812 11:29:04.760866 15315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:36515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.12.110.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.12.110.195
--webserver_port=0
--tserver_master_addrs=127.12.110.254:41681
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.12.110.195
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:04.762214 15315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:04.763985 15315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:04.777532 15323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:04.777508 15324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:04.777592 15327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:04.778496 15315 server_base.cc:1034] running on GCE node
I20260812 11:29:04.779282 15315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:04.780279 15315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:04.781589 15315 hybrid_clock.cc:648] HybridClock initialized: now 1786534144781477 us; error 97 us; skew 500 ppm
I20260812 11:29:04.781940 15315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260812 11:29:04.784260 15315 webserver.cc:460] Webserver started at http://127.12.110.195:46085/ using document root <none> and password file <none>
I20260812 11:29:04.784963 15315 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:04.785079 15315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:04.785394 15315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:04.788288 15315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data/instance:
uuid: "3607112de941473e82f163fd0d54d0a2"
format_stamp: "Formatted at 2026-08-12 11:29:04 on dist-test-slave-h6n0"
server_key: "fd087abc4d7ee10fb32dceb12dc5421f"
server_key_iv: "ce6ec97e95e3e1e6b94636fd31c2fb8d"
server_key_version: "encryptionkey@0"
I20260812 11:29:04.789086 15315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance:
uuid: "3607112de941473e82f163fd0d54d0a2"
format_stamp: "Formatted at 2026-08-12 11:29:04 on dist-test-slave-h6n0"
server_key: "fd087abc4d7ee10fb32dceb12dc5421f"
server_key_iv: "ce6ec97e95e3e1e6b94636fd31c2fb8d"
server_key_version: "encryptionkey@0"
I20260812 11:29:04.793982 15315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260812 11:29:04.797474 15336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:04.799041 15315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:29:04.799233 15315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal
uuid: "3607112de941473e82f163fd0d54d0a2"
format_stamp: "Formatted at 2026-08-12 11:29:04 on dist-test-slave-h6n0"
server_key: "fd087abc4d7ee10fb32dceb12dc5421f"
server_key_iv: "ce6ec97e95e3e1e6b94636fd31c2fb8d"
server_key_version: "encryptionkey@0"
I20260812 11:29:04.799463 15315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:04.823812 15315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:04.824615 15315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:04.824919 15315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:04.825937 15315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260812 11:29:04.827420 15315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260812 11:29:04.827543 15315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:04.827664 15315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260812 11:29:04.827757 15315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:04.863523 15315 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.195:46549
I20260812 11:29:04.863673 15520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.195:46549 every 8 connection(s)
I20260812 11:29:04.865198 15315 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/data/info.pb
I20260812 11:29:04.872630 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15315
I20260812 11:29:04.872846 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal/instance
I20260812 11:29:04.873381 12731 external_mini_cluster.cc:1428] Setting key d72250966754cb259907e49b07ef6835
I20260812 11:29:04.880966 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.868307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.195:45805 (local address 127.12.110.254:41681)
0812 11:29:04.868521 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:04.868527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:04.869969 (+ 1442us) server_negotiation.cc:407] Connection header received
0812 11:29:04.872282 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:04.872287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:04.872374 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:04.872467 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:04.873920 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.874782 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.877281 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.877702 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.877806 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.880003 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:04.880010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:04.880015 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:04.880037 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:04.880080 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:04.880105 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:04.880216 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:04.880465 (+ 249us) server_negotiation.cc:299] Negotiation successful
0812 11:29:04.880567 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260812 11:29:04.881995 15523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.868745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:41681 (local address 127.12.110.195:45805)
0812 11:29:04.869795 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:04.869829 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:04.871980 (+ 2151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:04.872731 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:04.872744 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:04.872868 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:04.873707 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.873735 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.874947 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.874956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.877126 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.877142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.877949 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.877957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.878281 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.879510 (+ 1229us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0812 11:29:04.879513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:04.879559 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0812 11:29:04.879701 (+ 142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:04.880215 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:04.880228 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:04.880239 (+ 11us) client_negotiation.cc:769] Sending connection context
0812 11:29:04.880437 (+ 198us) client_negotiation.cc:240] Negotiation successful
0812 11:29:04.880835 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":421,"threads_started":1}
I20260812 11:29:04.883471 15521 heartbeater.cc:347] Connected to a master server at 127.12.110.254:41681
I20260812 11:29:04.883800 15521 heartbeater.cc:464] Registering TS with master...
I20260812 11:29:04.884521 15521 heartbeater.cc:511] Master 127.12.110.254:41681 requested a full tablet report, sending...
I20260812 11:29:04.885905 14791 ts_manager.cc:194] Registered new tserver with Master: 3607112de941473e82f163fd0d54d0a2 (127.12.110.195:46549)
I20260812 11:29:04.886813 14791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.12.110.195:45805
I20260812 11:29:04.894019 12731 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260812 11:29:04.904430 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.898464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32888 (local address 127.12.110.254:41681)
0812 11:29:04.898821 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:04.898829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:04.899192 (+ 363us) server_negotiation.cc:407] Connection header received
0812 11:29:04.899392 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:04.899397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:04.899491 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:04.899600 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:04.900097 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.900774 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.902119 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.902566 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.902709 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.903726 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:04.903730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:04.903733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:04.903749 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:04.903777 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:04.903789 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:04.903888 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:04.904053 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 11:29:04.904143 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260812 11:29:04.908106 15104 heartbeater.cc:503] Master 127.12.110.254:41681 was elected leader, sending a full tablet report...
I20260812 11:29:04.907840 14791 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260812 11:29:04.914938 14791 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260812 11:29:04.943557 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.927126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41138 (local address 127.12.110.194:35113)
0812 11:29:04.932255 (+ 5129us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:04.932269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:04.933142 (+ 873us) server_negotiation.cc:407] Connection header received
0812 11:29:04.933283 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:04.933295 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:04.933631 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:04.935413 (+ 1782us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:04.936857 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.938262 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.940665 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.942280 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.942469 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.942737 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:04.943032 (+ 295us) server_negotiation.cc:299] Negotiation successful
0812 11:29:04.943201 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4911,"thread_start_us":4688,"threads_started":1}
I20260812 11:29:04.944154 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.927078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.194:35113 (local address 127.0.0.1:41138)
0812 11:29:04.932790 (+ 5712us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:04.932813 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:04.932975 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:04.935773 (+ 2798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:04.935784 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:04.935824 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:04.936608 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.936622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.938429 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.938441 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.940427 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.940443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.942481 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.942487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.942727 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.942791 (+ 64us) client_negotiation.cc:769] Sending connection context
0812 11:29:04.942982 (+ 191us) client_negotiation.cc:240] Negotiation successful
0812 11:29:04.943174 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5546,"thread_start_us":5394,"threads_started":1}
I20260812 11:29:04.946065 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.929031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:46549 (local address 127.0.0.1:47876)
0812 11:29:04.936757 (+ 7726us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:04.936778 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:04.937006 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:04.937994 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:04.937998 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:04.938029 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:04.938518 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.938576 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.940598 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.940604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.942271 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.942290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.945273 (+ 2983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.945277 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.945410 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.945469 (+ 59us) client_negotiation.cc:769] Sending connection context
0812 11:29:04.945572 (+ 103us) client_negotiation.cc:240] Negotiation successful
0812 11:29:04.945717 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7572,"mutex_wait_us":1,"thread_start_us":6902,"threads_started":1}
I20260812 11:29:04.946146 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.929258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47876 (local address 127.12.110.195:46549)
0812 11:29:04.935939 (+ 6681us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:04.935957 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:04.936927 (+ 970us) server_negotiation.cc:407] Connection header received
0812 11:29:04.937317 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:04.937328 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:04.937723 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:04.937966 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:04.938822 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.940444 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.942524 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.945074 (+ 2550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.945279 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.945488 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:04.945631 (+ 143us) server_negotiation.cc:299] Negotiation successful
0812 11:29:04.945801 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6355,"thread_start_us":6206,"threads_started":1}
I20260812 11:29:04.947445 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.926654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52416)
0812 11:29:04.929710 (+ 3056us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:04.929759 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:04.931454 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:04.937090 (+ 5636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:04.937098 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:04.937129 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:04.937621 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.937635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.939643 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.939649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.941935 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:04.942049 (+ 114us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.946561 (+ 4512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.946568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:04.946801 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.946835 (+ 34us) client_negotiation.cc:769] Sending connection context
0812 11:29:04.946909 (+ 74us) client_negotiation.cc:240] Negotiation successful
0812 11:29:04.947044 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2727,"mutex_wait_us":25,"thread_start_us":3389,"threads_started":1}
I20260812 11:29:04.948205 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:04.926912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.12.110.193:36711)
0812 11:29:04.936448 (+ 9536us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:04.936456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:04.936486 (+ 30us) server_negotiation.cc:407] Connection header received
0812 11:29:04.936580 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:04.936587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:04.936817 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:04.937065 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:04.937926 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.939493 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.942322 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:04.946289 (+ 3967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:04.947301 (+ 1012us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:04.947522 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:04.947594 (+ 72us) server_negotiation.cc:299] Negotiation successful
0812 11:29:04.947801 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9308,"thread_start_us":9202,"threads_started":1}
I20260812 11:29:04.948570 15208 tablet_service.cc:1467] Processing CreateTablet for tablet 9d06253f7b124b449a029d52c85e2880 (DEFAULT_TABLE table=test-table [id=c4597048f7b247c8b12d35e6794947df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:29:04.949554 15419 tablet_service.cc:1467] Processing CreateTablet for tablet 9d06253f7b124b449a029d52c85e2880 (DEFAULT_TABLE table=test-table [id=c4597048f7b247c8b12d35e6794947df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:29:04.950129 15208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d06253f7b124b449a029d52c85e2880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:04.950901 15419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d06253f7b124b449a029d52c85e2880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:04.951092 14992 tablet_service.cc:1467] Processing CreateTablet for tablet 9d06253f7b124b449a029d52c85e2880 (DEFAULT_TABLE table=test-table [id=c4597048f7b247c8b12d35e6794947df]), partition=RANGE (key) PARTITION UNBOUNDED
I20260812 11:29:04.952291 14992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d06253f7b124b449a029d52c85e2880. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:04.962877 15548 tablet_bootstrap.cc:492] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: Bootstrap starting.
I20260812 11:29:04.965243 15549 tablet_bootstrap.cc:492] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Bootstrap starting.
I20260812 11:29:04.967090 15548 tablet_bootstrap.cc:654] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:04.968426 15548 log.cc:824] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:04.968704 15549 tablet_bootstrap.cc:654] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:04.969898 15549 log.cc:824] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:04.970614 15550 tablet_bootstrap.cc:492] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Bootstrap starting.
I20260812 11:29:04.971897 15548 tablet_bootstrap.cc:492] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: No bootstrap required, opened a new log
I20260812 11:29:04.972215 15548 ts_tablet_manager.cc:1397] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260812 11:29:04.973474 15550 tablet_bootstrap.cc:654] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:04.974381 15550 log.cc:824] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:04.978790 15549 tablet_bootstrap.cc:492] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: No bootstrap required, opened a new log
I20260812 11:29:04.979043 15549 ts_tablet_manager.cc:1397] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260812 11:29:04.980633 15550 tablet_bootstrap.cc:492] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: No bootstrap required, opened a new log
I20260812 11:29:04.980907 15550 ts_tablet_manager.cc:1397] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260812 11:29:04.983785 15548 raft_consensus.cc:348] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:04.984215 15548 raft_consensus.cc:374] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:04.984313 15548 raft_consensus.cc:729] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33f71f1d8d284065a5b103b595a8021f, State: Initialized, Role: FOLLOWER
I20260812 11:29:04.984933 15548 consensus_queue.cc:260] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:04.987192 15548 ts_tablet_manager.cc:1428] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260812 11:29:04.989329 15550 raft_consensus.cc:348] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:04.989302 15549 raft_consensus.cc:348] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:04.989684 15550 raft_consensus.cc:374] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:04.989737 15549 raft_consensus.cc:374] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:04.989813 15550 raft_consensus.cc:729] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f8aaf7128463ea9a2446d036c31df, State: Initialized, Role: FOLLOWER
I20260812 11:29:04.989845 15549 raft_consensus.cc:729] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3607112de941473e82f163fd0d54d0a2, State: Initialized, Role: FOLLOWER
I20260812 11:29:04.990407 15550 consensus_queue.cc:260] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:04.990581 15549 consensus_queue.cc:260] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:04.991674 15312 heartbeater.cc:503] Master 127.12.110.254:41681 was elected leader, sending a full tablet report...
I20260812 11:29:04.992244 15550 ts_tablet_manager.cc:1428] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260812 11:29:04.993219 15521 heartbeater.cc:503] Master 127.12.110.254:41681 was elected leader, sending a full tablet report...
I20260812 11:29:04.993790 15549 ts_tablet_manager.cc:1428] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260812 11:29:05.052138 15561 raft_consensus.cc:482] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260812 11:29:05.052456 15561 raft_consensus.cc:504] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:05.054246 15561 leader_election.cc:288] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33f71f1d8d284065a5b103b595a8021f (127.12.110.193:36711), 3607112de941473e82f163fd0d54d0a2 (127.12.110.195:46549)
I20260812 11:29:05.061407 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.054614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:42205 (local address 127.12.110.193:36711)
0812 11:29:05.054819 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:05.054827 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:05.055598 (+ 771us) server_negotiation.cc:407] Connection header received
0812 11:29:05.055791 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:05.055797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:05.055891 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:05.055987 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:05.056664 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.057403 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.059686 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.060466 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.060649 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.060720 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:05.061041 (+ 321us) server_negotiation.cc:299] Negotiation successful
0812 11:29:05.061168 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:29:05.061573 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.054466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.12.110.194:42205)
0812 11:29:05.055457 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.055479 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:05.055638 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:05.056037 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:05.056042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:05.056066 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:05.056493 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:05.056507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.057561 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.057566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:05.059478 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:05.059500 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.060672 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.060679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:05.060847 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.060897 (+ 50us) client_negotiation.cc:769] Sending connection context
0812 11:29:05.061003 (+ 106us) client_negotiation.cc:240] Negotiation successful
0812 11:29:05.061124 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":9,"thread_start_us":562,"threads_started":1}
I20260812 11:29:05.061630 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.054765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.195:46549 (local address 127.12.110.194:51025)
0812 11:29:05.055632 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.055657 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:05.055805 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:05.056238 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:05.056242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:05.056264 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0812 11:29:05.056676 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:05.056687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.057746 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.057751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:05.059545 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:05.059558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.060704 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.060710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:05.060905 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.060927 (+ 22us) client_negotiation.cc:769] Sending connection context
0812 11:29:05.061034 (+ 107us) client_negotiation.cc:240] Negotiation successful
0812 11:29:05.061160 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":23,"thread_start_us":535,"threads_started":1}
I20260812 11:29:05.061430 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.054959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.12.110.194:51025 (local address 127.12.110.195:46549)
0812 11:29:05.055151 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:05.055157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:05.055783 (+ 626us) server_negotiation.cc:407] Connection header received
0812 11:29:05.056013 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:05.056018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:05.056100 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:05.056214 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0812 11:29:05.056839 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.057571 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.059735 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.060516 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.060692 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.060758 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:05.061058 (+ 300us) server_negotiation.cc:299] Negotiation successful
0812 11:29:05.061180 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:29:05.062678 15026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d06253f7b124b449a029d52c85e2880" candidate_uuid: "444f8aaf7128463ea9a2446d036c31df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f71f1d8d284065a5b103b595a8021f" is_pre_election: true
I20260812 11:29:05.062775 15452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d06253f7b124b449a029d52c85e2880" candidate_uuid: "444f8aaf7128463ea9a2446d036c31df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3607112de941473e82f163fd0d54d0a2" is_pre_election: true
I20260812 11:29:05.063226 15026 raft_consensus.cc:2454] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 444f8aaf7128463ea9a2446d036c31df in term 0.
I20260812 11:29:05.063288 15452 raft_consensus.cc:2454] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 444f8aaf7128463ea9a2446d036c31df in term 0.
I20260812 11:29:05.063896 15135 leader_election.cc:302] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33f71f1d8d284065a5b103b595a8021f, 444f8aaf7128463ea9a2446d036c31df; no voters:
I20260812 11:29:05.064281 15561 raft_consensus.cc:2781] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260812 11:29:05.064437 15561 raft_consensus.cc:482] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260812 11:29:05.064550 15561 raft_consensus.cc:3037] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:05.066744 15561 raft_consensus.cc:504] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:05.067310 15561 leader_election.cc:288] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [CANDIDATE]: Term 1 election: Requested vote from peers 33f71f1d8d284065a5b103b595a8021f (127.12.110.193:36711), 3607112de941473e82f163fd0d54d0a2 (127.12.110.195:46549)
I20260812 11:29:05.067644 15026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d06253f7b124b449a029d52c85e2880" candidate_uuid: "444f8aaf7128463ea9a2446d036c31df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33f71f1d8d284065a5b103b595a8021f"
I20260812 11:29:05.067850 15026 raft_consensus.cc:3037] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:05.067821 15452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d06253f7b124b449a029d52c85e2880" candidate_uuid: "444f8aaf7128463ea9a2446d036c31df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3607112de941473e82f163fd0d54d0a2"
I20260812 11:29:05.068022 15452 raft_consensus.cc:3037] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:05.070215 15452 raft_consensus.cc:2454] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 444f8aaf7128463ea9a2446d036c31df in term 1.
I20260812 11:29:05.070225 15026 raft_consensus.cc:2454] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 444f8aaf7128463ea9a2446d036c31df in term 1.
I20260812 11:29:05.070659 15130 leader_election.cc:302] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3607112de941473e82f163fd0d54d0a2, 444f8aaf7128463ea9a2446d036c31df; no voters:
I20260812 11:29:05.070993 15561 raft_consensus.cc:2781] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:05.071395 15561 raft_consensus.cc:686] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 1 LEADER]: Becoming Leader. State: Replica: 444f8aaf7128463ea9a2446d036c31df, State: Running, Role: LEADER
I20260812 11:29:05.071897 15561 consensus_queue.cc:237] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } }
I20260812 11:29:05.076241 14790 catalog_manager.cc:5495] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df reported cstate change: term changed from 0 to 1, leader changed from <none> to 444f8aaf7128463ea9a2446d036c31df (127.12.110.194). New cstate: current_term: 1 leader_uuid: "444f8aaf7128463ea9a2446d036c31df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "444f8aaf7128463ea9a2446d036c31df" member_type: VOTER last_known_addr { host: "127.12.110.194" port: 35113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 } health_report { overall_health: UNKNOWN } } }
I20260812 11:29:05.100984 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.095454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41148 (local address 127.12.110.194:35113)
0812 11:29:05.095665 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:05.095671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:05.095794 (+ 123us) server_negotiation.cc:407] Connection header received
0812 11:29:05.096052 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:05.096062 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:05.096153 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:05.096309 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:05.096691 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.097434 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.098850 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.099273 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.099418 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.099871 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:05.100325 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:05.100500 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:05.100564 (+ 64us) server_negotiation.cc:299] Negotiation successful
0812 11:29:05.100640 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:29:05.109225 15452 raft_consensus.cc:1261] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 1 FOLLOWER]: Refusing update from remote peer 444f8aaf7128463ea9a2446d036c31df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:05.109246 15026 raft_consensus.cc:1261] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 1 FOLLOWER]: Refusing update from remote peer 444f8aaf7128463ea9a2446d036c31df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:05.110375 15561 consensus_queue.cc:1035] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [LEADER]: Connected to new peer: Peer: permanent_uuid: "3607112de941473e82f163fd0d54d0a2" member_type: VOTER last_known_addr { host: "127.12.110.195" port: 46549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:29:05.110751 15567 consensus_queue.cc:1035] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [LEADER]: Connected to new peer: Peer: permanent_uuid: "33f71f1d8d284065a5b103b595a8021f" member_type: VOTER last_known_addr { host: "127.12.110.193" port: 36711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260812 11:29:05.117244 15522 tablet.cc:2369] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:29:05.125231 15576 mvcc.cc:205] Tried to move back new op lower bound from 7317643858356957184 to 7317643858218283008. Current Snapshot: MvccSnapshot[applied={T|T < 7317643858356957184}]
I20260812 11:29:05.136901 14790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:29:05.137233 14790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:32888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260812 11:29:05.142089 14790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2edfb13f3ff4832936f2674975a3dbb: Sending DeleteTablet for 3 replicas of tablet 9d06253f7b124b449a029d52c85e2880
I20260812 11:29:05.143267 15208 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d06253f7b124b449a029d52c85e2880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:29:05 UTC) from {username='slave'} at 127.0.0.1:41138
I20260812 11:29:05.143260 15419 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d06253f7b124b449a029d52c85e2880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:29:05 UTC) from {username='slave'} at 127.0.0.1:47876
I20260812 11:29:05.143364 14992 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d06253f7b124b449a029d52c85e2880 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-12 11:29:05 UTC) from {username='slave'} at 127.0.0.1:52416
I20260812 11:29:05.145851 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14890
I20260812 11:29:05.153019 15591 tablet_replica.cc:330] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: stopping tablet replica
I20260812 11:29:05.153682 15591 raft_consensus.cc:2229] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:29:05.154395 15591 raft_consensus.cc:2258] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:29:05.155112 15588 tablet_replica.cc:330] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: stopping tablet replica
I20260812 11:29:05.155845 15588 raft_consensus.cc:2229] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260812 11:29:05.158454 15591 ts_tablet_manager.cc:1905] T 9d06253f7b124b449a029d52c85e2880 P 33f71f1d8d284065a5b103b595a8021f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:29:05.159001 15588 raft_consensus.cc:2258] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:29:05.159658 15589 tablet_replica.cc:330] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: stopping tablet replica
I20260812 11:29:05.160079 15589 raft_consensus.cc:2229] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 1 LEADER]: Raft consensus shutting down.
I20260812 11:29:05.160751 15589 raft_consensus.cc:2258] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260812 11:29:05.161945 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.147312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.12.110.193:36711)
0812 11:29:05.147559 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:05.147569 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:05.152972 (+ 5403us) server_negotiation.cc:407] Connection header received
0812 11:29:05.153464 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:05.153471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:05.153614 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:05.153714 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:05.154502 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.155706 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.157959 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.158919 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.159080 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.160518 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:05.160526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:05.160538 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:05.161142 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:05.161188 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:05.161207 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:05.161354 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:05.161448 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 11:29:05.161555 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:29:05.162734 15097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:05.165242 15588 ts_tablet_manager.cc:1905] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:29:05.168503 15589 ts_tablet_manager.cc:1905] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260812 11:29:05.174494 15588 ts_tablet_manager.cc:1918] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:29:05.174722 15588 log.cc:1196] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-2/wal/wals/9d06253f7b124b449a029d52c85e2880
I20260812 11:29:05.175375 15588 ts_tablet_manager.cc:1939] T 9d06253f7b124b449a029d52c85e2880 P 3607112de941473e82f163fd0d54d0a2: Deleting consensus metadata
I20260812 11:29:05.176363 15589 ts_tablet_manager.cc:1918] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260812 11:29:05.176616 15589 log.cc:1196] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Deleting WAL directory at /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786533802845873-12731-0/minicluster-data/ts-1/wal/wals/9d06253f7b124b449a029d52c85e2880
I20260812 11:29:05.177270 15589 ts_tablet_manager.cc:1939] T 9d06253f7b124b449a029d52c85e2880 P 444f8aaf7128463ea9a2446d036c31df: Deleting consensus metadata
I20260812 11:29:05.177640 14760 catalog_manager.cc:4841] TS 3607112de941473e82f163fd0d54d0a2 (127.12.110.195:46549): tablet 9d06253f7b124b449a029d52c85e2880 (table test-table [id=c4597048f7b247c8b12d35e6794947df]) successfully deleted
I20260812 11:29:05.179217 14766 catalog_manager.cc:4841] TS 444f8aaf7128463ea9a2446d036c31df (127.12.110.194:35113): tablet 9d06253f7b124b449a029d52c85e2880 (table test-table [id=c4597048f7b247c8b12d35e6794947df]) successfully deleted
I20260812 11:29:05.269186 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14890
I20260812 11:29:05.296109 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15109
W20260812 11:29:05.296099 14769 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.12.110.193:36711 (error 108)
I20260812 11:29:05.298441 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.297877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52434)
0812 11:29:05.298130 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.298259 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260812 11:29:05.298724 14769 catalog_manager.cc:4584] TS 33f71f1d8d284065a5b103b595a8021f (127.12.110.193:36711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9d06253f7b124b449a029d52c85e2880: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
I20260812 11:29:05.304714 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.297549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.12.110.194:35113)
0812 11:29:05.297750 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:05.297756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:05.297775 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:29:05.297878 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:05.297883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:05.297991 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:05.298106 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:05.298838 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.299746 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.300973 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.301574 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.301711 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.302987 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:05.302996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:05.303009 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:05.303726 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:05.303772 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:05.303899 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:05.304036 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:05.304150 (+ 114us) server_negotiation.cc:299] Negotiation successful
0812 11:29:05.304249 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:29:05.305469 15305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:05.352629 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.352126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52444)
0812 11:29:05.352325 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.352422 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260812 11:29:05.353785 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.353444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52460)
0812 11:29:05.353589 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.353666 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260812 11:29:05.414971 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.414077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52474)
0812 11:29:05.414506 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.414698 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262}
I20260812 11:29:05.416257 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.415856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52482)
0812 11:29:05.416028 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.416117 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260812 11:29:05.422413 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15109
I20260812 11:29:05.449681 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15315
I20260812 11:29:05.456583 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.450468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.12.110.195:46549)
0812 11:29:05.450686 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:05.450691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:05.450709 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:29:05.450787 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:05.450791 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:05.450871 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:05.450957 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:05.451359 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.452232 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.453646 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:05.454041 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:05.454137 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:05.455288 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:05.455296 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:05.455310 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:05.455862 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:05.455907 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:05.455927 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:05.456029 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:05.456160 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 11:29:05.456255 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260812 11:29:05.457267 15512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:05.497807 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.497293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52494)
0812 11:29:05.497529 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.497631 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260812 11:29:05.498967 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.498607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52500)
0812 11:29:05.498794 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.498863 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260812 11:29:05.547586 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15315
I20260812 11:29:05.577654 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 14735
I20260812 11:29:05.578903 14845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:05.690707 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:05.689969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.193:36711 (local address 127.0.0.1:52504)
0812 11:29:05.690217 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:05.690361 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.193:36711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 11:29:05.690919 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 14735
2026-08-12T11:29:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2949 ms)
[----------] 6 tests from EncryptionPolicyTest (23847 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:05 dist-test-slave-h6n0 krb5kdc[15604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:05 dist-test-slave-h6n0 krb5kdc[15604](info): set up 2 sockets
Aug 12 11:29:05 dist-test-slave-h6n0 krb5kdc[15604](info): commencing operation
krb5kdc: starting...
W20260812 11:29:07.083329 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.342s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:07 dist-test-slave-h6n0 krb5kdc[15604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:07.437949 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:07.802337 15626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:07.802721 15626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:07.802824 15626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:07.809993 15626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:07.810092 15626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:07.810138 15626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:07.810200 15626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:07.810245 15626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:07.819826 15626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:07.820999 15626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:07.822188 15626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:07.834434 15638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:07.834597 15633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:07.834581 15636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:07.835413 15626 server_base.cc:1034] running on GCE node
I20260812 11:29:07.836228 15626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:07.837598 15626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:07.838798 15626 hybrid_clock.cc:648] HybridClock initialized: now 1786534147838754 us; error 38 us; skew 500 ppm
I20260812 11:29:07.839172 15626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:07 dist-test-slave-h6n0 krb5kdc[15604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:07.847509 15626 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:07.849123 15626 webserver.cc:460] Webserver started at http://127.0.0.1:37555/ using document root <none> and password file <none>
I20260812 11:29:07.849807 15626 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:07.849967 15626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:07.850289 15626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:07.853550 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "f309df2a93744029843493062d042f0b"
format_stamp: "Formatted at 2026-08-12 11:29:07 on dist-test-slave-h6n0"
server_key: "b6e060e6cbeaed83ee290e9d328aaf9f"
server_key_iv: "d984e4086142a90d67c828c17ab5ab3e"
server_key_version: "encryptionkey@0"
I20260812 11:29:07.854348 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "f309df2a93744029843493062d042f0b"
format_stamp: "Formatted at 2026-08-12 11:29:07 on dist-test-slave-h6n0"
server_key: "b6e060e6cbeaed83ee290e9d328aaf9f"
server_key_iv: "d984e4086142a90d67c828c17ab5ab3e"
server_key_version: "encryptionkey@0"
I20260812 11:29:07.859630 15626 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:29:07.863516 15649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:07.864882 15626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:29:07.865096 15626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "f309df2a93744029843493062d042f0b"
format_stamp: "Formatted at 2026-08-12 11:29:07 on dist-test-slave-h6n0"
server_key: "b6e060e6cbeaed83ee290e9d328aaf9f"
server_key_iv: "d984e4086142a90d67c828c17ab5ab3e"
server_key_version: "encryptionkey@0"
I20260812 11:29:07.865312 15626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:07.890755 15626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:07.895349 15626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:07.895665 15626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:07.915323 15626 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39121
I20260812 11:29:07.915370 15739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39121 every 8 connection(s)
I20260812 11:29:07.916733 15626 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:07.920359 15740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:07.924726 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15626
I20260812 11:29:07.924954 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:07.925657 12731 external_mini_cluster.cc:1428] Setting key 9cca4acce1c0c7a9c40324b718a085b5
I20260812 11:29:07.933112 15740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b: Bootstrap starting.
I20260812 11:29:07.937062 15740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:07.938261 15740 log.cc:824] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:07.942040 15740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b: No bootstrap required, opened a new log
Aug 12 11:29:07 dist-test-slave-h6n0 krb5kdc[15604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534147, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:29:07.953634 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:07.927732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59669 (local address 127.0.0.1:39121)
0812 11:29:07.928561 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:07.928577 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:07.928627 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:29:07.931392 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:07.931423 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:07.931962 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:07.932511 (+ 549us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:07.934100 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:07.936197 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:07.937809 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:07.938964 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:07.939150 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:07.944306 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:07.944317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:07.944343 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:07.944402 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:07.948864 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:07.949720 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:07.949733 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:07.949741 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:07.949956 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:07.950419 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:07.950426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:07.950429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:07.951209 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:07.951568 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:07.951928 (+ 360us) server_negotiation.cc:299] Negotiation successful
0812 11:29:07.952287 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":363,"threads_started":1}
I20260812 11:29:07.953321 15740 raft_consensus.cc:348] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f309df2a93744029843493062d042f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260812 11:29:07.953809 15740 raft_consensus.cc:374] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:07.953938 15740 raft_consensus.cc:729] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f309df2a93744029843493062d042f0b, State: Initialized, Role: FOLLOWER
I20260812 11:29:07.954643 15740 consensus_queue.cc:260] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f309df2a93744029843493062d042f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260812 11:29:07.954941 15740 raft_consensus.cc:388] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:07.955112 15740 raft_consensus.cc:482] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:07.955307 15740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:07.958061 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:07.958138 15740 raft_consensus.cc:504] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f309df2a93744029843493062d042f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260812 11:29:07.958761 15740 leader_election.cc:302] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f309df2a93744029843493062d042f0b; no voters:
I20260812 11:29:07.959568 15740 leader_election.cc:288] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:07.959911 15748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:07.961210 15748 raft_consensus.cc:686] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [term 1 LEADER]: Becoming Leader. State: Replica: f309df2a93744029843493062d042f0b, State: Running, Role: LEADER
I20260812 11:29:07.961815 15740 sys_catalog.cc:564] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:07.961908 15748 consensus_queue.cc:237] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f309df2a93744029843493062d042f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } }
I20260812 11:29:07.965715 15753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f309df2a93744029843493062d042f0b. Latest consensus state: current_term: 1 leader_uuid: "f309df2a93744029843493062d042f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f309df2a93744029843493062d042f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260812 11:29:07.966202 15753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:07.966903 15751 sys_catalog.cc:455] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f309df2a93744029843493062d042f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f309df2a93744029843493062d042f0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39121 } } }
I20260812 11:29:07.967751 15751 sys_catalog.cc:458] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:07.976732 15762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:07.983104 15762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:07.993907 15762 catalog_manager.cc:1347] Generated new cluster ID: 1a886d9cafcb46c3a3c7e65c25766cfb
I20260812 11:29:07.994060 15762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:07.999447 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:07.978351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42365 (local address 127.0.0.1:39121)
0812 11:29:07.978591 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:07.978600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:07.979465 (+ 865us) server_negotiation.cc:407] Connection header received
0812 11:29:07.979805 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:07.979814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:07.979926 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:07.980040 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:07.982718 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:07.984313 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:07.986840 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:07.987752 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:07.987884 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:07.991451 (+ 3567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:07.991458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:07.991463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:07.991512 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:07.995706 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:07.996636 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:07.996642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:07.996646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:07.996819 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:07.997330 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:07.997335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:07.997338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:07.997799 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:07.998100 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:07.998474 (+ 374us) server_negotiation.cc:299] Negotiation successful
0812 11:29:07.998710 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 11:29:08.008486 15762 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:08.010303 15762 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:08.017912 15762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f309df2a93744029843493062d042f0b: Generated new TSK 0
I20260812 11:29:08.018911 15762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:08.030740 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15626
I20260812 11:29:08.032869 15734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:08.122255 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15626
W20260812 11:29:08.147181 15602 connection.cc:535] client connection to 127.0.0.1:39121 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39121): Connection reset by peer (error 104)
2026-08-12T11:29:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2426 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:08 dist-test-slave-h6n0 krb5kdc[15790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:08 dist-test-slave-h6n0 krb5kdc[15790](info): set up 2 sockets
Aug 12 11:29:08 dist-test-slave-h6n0 krb5kdc[15790](info): commencing operation
krb5kdc: starting...
W20260812 11:29:09.533762 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.367s user 0.002s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:09 dist-test-slave-h6n0 krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:09.873687 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:10.224511 15807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:10.224838 15807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:10.224941 15807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:10.232594 15807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:10.232708 15807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:10.232757 15807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:10.232831 15807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:10.232883 15807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:10.243306 15807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41955
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:10.244546 15807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:10.245810 15807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:10.257400 15817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:10.257490 15825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:10.257514 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:10.259075 15807 server_base.cc:1034] running on GCE node
I20260812 11:29:10.259858 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:10.261315 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:10.262614 15807 hybrid_clock.cc:648] HybridClock initialized: now 1786534150262546 us; error 43 us; skew 500 ppm
I20260812 11:29:10.263013 15807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:10 dist-test-slave-h6n0 krb5kdc[15790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534150, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:10.266698 15807 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:10.268381 15807 webserver.cc:460] Webserver started at http://127.0.0.1:40081/ using document root <none> and password file <none>
I20260812 11:29:10.269057 15807 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:10.269207 15807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:10.269510 15807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:10.273411 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "c94744823ccd4ec08a7d52a56c02c88e"
format_stamp: "Formatted at 2026-08-12 11:29:10 on dist-test-slave-h6n0"
server_key: "a450b2f0425f4a28fc491a64cf622da4"
server_key_iv: "7556666088879eeae8d31b7b84bab751"
server_key_version: "encryptionkey@0"
I20260812 11:29:10.274523 15807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "c94744823ccd4ec08a7d52a56c02c88e"
format_stamp: "Formatted at 2026-08-12 11:29:10 on dist-test-slave-h6n0"
server_key: "a450b2f0425f4a28fc491a64cf622da4"
server_key_iv: "7556666088879eeae8d31b7b84bab751"
server_key_version: "encryptionkey@0"
I20260812 11:29:10.281672 15807 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260812 11:29:10.285149 15837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:10.286432 15807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 11:29:10.286659 15807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "c94744823ccd4ec08a7d52a56c02c88e"
format_stamp: "Formatted at 2026-08-12 11:29:10 on dist-test-slave-h6n0"
server_key: "a450b2f0425f4a28fc491a64cf622da4"
server_key_iv: "7556666088879eeae8d31b7b84bab751"
server_key_version: "encryptionkey@0"
I20260812 11:29:10.286859 15807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:10.301014 15807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:10.305510 15807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:10.305814 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:10.326756 15807 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41955
I20260812 11:29:10.326793 15919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41955 every 8 connection(s)
I20260812 11:29:10.328177 15807 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:10.331840 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:10.334488 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15807
I20260812 11:29:10.334692 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:10.335260 12731 external_mini_cluster.cc:1428] Setting key 8e7a98da68756002d663304ee548078e
I20260812 11:29:10.344146 15921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e: Bootstrap starting.
I20260812 11:29:10.347823 15921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:10.348919 15921 log.cc:824] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:10.349040 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:10.337041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58951 (local address 127.0.0.1:41955)
0812 11:29:10.337881 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:10.337895 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:10.337937 (+ 42us) server_negotiation.cc:407] Connection header received
0812 11:29:10.339686 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:10.339706 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:10.340064 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:10.340358 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:10.340982 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:10.342295 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:10.344200 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:10.345004 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:10.345151 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:10.346487 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:10.346495 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:10.346507 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:10.347080 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:10.347152 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:10.347176 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:10.347291 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:10.347883 (+ 592us) server_negotiation.cc:299] Negotiation successful
0812 11:29:10.348041 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":390,"threads_started":1}
I20260812 11:29:10.351953 15921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e: No bootstrap required, opened a new log
I20260812 11:29:10.353291 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:10.360257 15921 raft_consensus.cc:348] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94744823ccd4ec08a7d52a56c02c88e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260812 11:29:10.360644 15921 raft_consensus.cc:374] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:10.360762 15921 raft_consensus.cc:729] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94744823ccd4ec08a7d52a56c02c88e, State: Initialized, Role: FOLLOWER
I20260812 11:29:10.361392 15921 consensus_queue.cc:260] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94744823ccd4ec08a7d52a56c02c88e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260812 11:29:10.361663 15921 raft_consensus.cc:388] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:10.361821 15921 raft_consensus.cc:482] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:10.362085 15921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:10.364526 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:10.357953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39531 (local address 127.0.0.1:41955)
0812 11:29:10.358171 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:10.358183 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:10.358567 (+ 384us) server_negotiation.cc:407] Connection header received
0812 11:29:10.358790 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:10.358794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:10.358858 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:10.358982 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:10.359531 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:10.360279 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:10.361799 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:10.362368 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:10.362509 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:10.363736 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:10.363742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:10.363746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:10.363773 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:10.363828 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:10.363844 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:10.363970 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:10.364175 (+ 205us) server_negotiation.cc:299] Negotiation successful
0812 11:29:10.364230 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260812 11:29:10.364523 15921 raft_consensus.cc:504] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94744823ccd4ec08a7d52a56c02c88e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260812 11:29:10.365198 15921 leader_election.cc:302] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c94744823ccd4ec08a7d52a56c02c88e; no voters:
I20260812 11:29:10.365876 15921 leader_election.cc:288] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:10.366007 15936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:10.366691 15936 raft_consensus.cc:686] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [term 1 LEADER]: Becoming Leader. State: Replica: c94744823ccd4ec08a7d52a56c02c88e, State: Running, Role: LEADER
I20260812 11:29:10.367336 15936 consensus_queue.cc:237] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94744823ccd4ec08a7d52a56c02c88e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } }
I20260812 11:29:10.368386 15921 sys_catalog.cc:564] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:10.370319 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c94744823ccd4ec08a7d52a56c02c88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94744823ccd4ec08a7d52a56c02c88e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } } }
I20260812 11:29:10.370772 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:10.370939 15938 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94744823ccd4ec08a7d52a56c02c88e. Latest consensus state: current_term: 1 leader_uuid: "c94744823ccd4ec08a7d52a56c02c88e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c94744823ccd4ec08a7d52a56c02c88e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41955 } } }
I20260812 11:29:10.371455 15938 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:10.375789 15941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:10.380628 15941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:10.389748 15941 catalog_manager.cc:1347] Generated new cluster ID: 10806c533f7444998cb27904665b9a43
I20260812 11:29:10.389854 15941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:10.409894 15941 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:10.411243 15941 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:10.420130 15941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c94744823ccd4ec08a7d52a56c02c88e: Generated new TSK 0
I20260812 11:29:10.420989 15941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:10.428345 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15807
W20260812 11:29:10.430354 15910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58951
W20260812 11:29:10.431344 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15807: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:10.431556 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15807
W20260812 11:29:10.459944 15787 connection.cc:535] client connection to 127.0.0.1:41955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41955): Connection reset by peer (error 104)
2026-08-12T11:29:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:10 dist-test-slave-h6n0 krb5kdc[15966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:10 dist-test-slave-h6n0 krb5kdc[15966](info): set up 2 sockets
Aug 12 11:29:10 dist-test-slave-h6n0 krb5kdc[15966](info): commencing operation
krb5kdc: starting...
W20260812 11:29:11.828914 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.350s user 0.001s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:12 dist-test-slave-h6n0 krb5kdc[15966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534152, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:12.162001 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:12.509891 15984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:12.510198 15984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:12.510368 15984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:12.517751 15984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:12.517863 15984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:12.517987 15984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:12.518113 15984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:12.518177 15984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:12.529515 15984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:12.531088 15984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:12.532577 15984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:12.545243 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:12.545204 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:12.545292 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:12.546036 15984 server_base.cc:1034] running on GCE node
I20260812 11:29:12.546890 15984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:12.548475 15984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:12.549705 15984 hybrid_clock.cc:648] HybridClock initialized: now 1786534152549654 us; error 34 us; skew 500 ppm
I20260812 11:29:12.550084 15984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:12 dist-test-slave-h6n0 krb5kdc[15966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:12.553611 15984 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:12.555508 15984 webserver.cc:460] Webserver started at http://127.0.0.1:33541/ using document root <none> and password file <none>
I20260812 11:29:12.556196 15984 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:12.556305 15984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:12.556646 15984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:12.559774 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8"
format_stamp: "Formatted at 2026-08-12 11:29:12 on dist-test-slave-h6n0"
server_key: "e3816e1e6f455eda90ddbd8f4cdda189"
server_key_iv: "337559b0d4ec8bc22d67906cf87afc6c"
server_key_version: "encryptionkey@0"
I20260812 11:29:12.560561 15984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8"
format_stamp: "Formatted at 2026-08-12 11:29:12 on dist-test-slave-h6n0"
server_key: "e3816e1e6f455eda90ddbd8f4cdda189"
server_key_iv: "337559b0d4ec8bc22d67906cf87afc6c"
server_key_version: "encryptionkey@0"
I20260812 11:29:12.565740 15984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:29:12.569342 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:12.570669 15984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:29:12.570883 15984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8"
format_stamp: "Formatted at 2026-08-12 11:29:12 on dist-test-slave-h6n0"
server_key: "e3816e1e6f455eda90ddbd8f4cdda189"
server_key_iv: "337559b0d4ec8bc22d67906cf87afc6c"
server_key_version: "encryptionkey@0"
I20260812 11:29:12.571220 15984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:12.577483 15984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:12.582578 15984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:12.582919 15984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:12.603504 15984 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40917
I20260812 11:29:12.603543 16093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40917 every 8 connection(s)
I20260812 11:29:12.605050 15984 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:12.608876 16094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:12.615000 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 15984
I20260812 11:29:12.615200 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:12.615760 12731 external_mini_cluster.cc:1428] Setting key c9ab4434456f74f0baf797a566f78ba3
I20260812 11:29:12.619877 16094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8: Bootstrap starting.
I20260812 11:29:12.623584 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:12.617572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57677 (local address 127.0.0.1:40917)
0812 11:29:12.618406 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:12.618418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:12.618458 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:29:12.620261 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:12.620284 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:12.620604 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:12.620907 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:12.622079 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:12.622085 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:12.622096 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:12.622111 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:12.622178 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:12.622199 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:12.622364 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:12.622470 (+ 106us) server_negotiation.cc:299] Negotiation successful
0812 11:29:12.622718 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":395,"threads_started":1}
I20260812 11:29:12.623641 16094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:12.624711 16094 log.cc:824] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:12.627389 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:12.627944 16094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8: No bootstrap required, opened a new log
I20260812 11:29:12.634431 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:12.631879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33213 (local address 127.0.0.1:40917)
0812 11:29:12.632103 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:12.632111 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:12.632497 (+ 386us) server_negotiation.cc:407] Connection header received
0812 11:29:12.632676 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:12.632680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:12.632728 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:12.632806 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:12.633744 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:12.633748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:12.633751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:12.633765 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:12.633819 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:12.633837 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:12.633931 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:12.634058 (+ 127us) server_negotiation.cc:299] Negotiation successful
0812 11:29:12.634101 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:29:12.636643 16094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260812 11:29:12.637017 16094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:12.637137 16094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268b9cac27fc4d0b99f76c6ee13aa5b8, State: Initialized, Role: FOLLOWER
I20260812 11:29:12.637694 16094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260812 11:29:12.637961 16094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:12.638115 16094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:12.638307 16094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:12.640545 16094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260812 11:29:12.641058 16094 leader_election.cc:302] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268b9cac27fc4d0b99f76c6ee13aa5b8; no voters:
I20260812 11:29:12.641625 16094 leader_election.cc:288] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:12.641729 16115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:12.642590 16115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [term 1 LEADER]: Becoming Leader. State: Replica: 268b9cac27fc4d0b99f76c6ee13aa5b8, State: Running, Role: LEADER
I20260812 11:29:12.643138 16115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } }
I20260812 11:29:12.643872 16094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:12.646046 16117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 268b9cac27fc4d0b99f76c6ee13aa5b8. Latest consensus state: current_term: 1 leader_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260812 11:29:12.646893 16116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268b9cac27fc4d0b99f76c6ee13aa5b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40917 } } }
I20260812 11:29:12.647287 16116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:12.647830 16117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:12.651021 16126 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:12.656930 16126 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:12.675120 16126 catalog_manager.cc:1347] Generated new cluster ID: 62dc3ac53c544c6a9b246b696c89eabb
I20260812 11:29:12.675251 16126 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:12.689322 16126 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:12.689786 16126 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:12.696898 16126 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 268b9cac27fc4d0b99f76c6ee13aa5b8: Generated new TSK 0
I20260812 11:29:12.697865 16126 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:12.707463 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15984
W20260812 11:29:12.709010 16086 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57677
W20260812 11:29:12.709766 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 15984: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:12.709949 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 15984
2026-08-12T11:29:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:12 dist-test-slave-h6n0 krb5kdc[16149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:12 dist-test-slave-h6n0 krb5kdc[16149](info): set up 2 sockets
Aug 12 11:29:12 dist-test-slave-h6n0 krb5kdc[16149](info): commencing operation
krb5kdc: starting...
W20260812 11:29:14.099272 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.343s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:14 dist-test-slave-h6n0 krb5kdc[16149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:14.433712 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:14.802803 16176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:14.803076 16176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:14.803267 16176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:14.810472 16176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:14.810653 16176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:14.810743 16176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:14.810859 16176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:14.810947 16176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:14.820640 16176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:14.822002 16176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:14.823331 16176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:14.835158 16185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:14.835314 16186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:14.835271 16189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:14.836325 16176 server_base.cc:1034] running on GCE node
I20260812 11:29:14.837123 16176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:14.838459 16176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:14.839732 16176 hybrid_clock.cc:648] HybridClock initialized: now 1786534154839681 us; error 43 us; skew 500 ppm
I20260812 11:29:14.840083 16176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:14 dist-test-slave-h6n0 krb5kdc[16149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:14.843246 16176 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:14.844748 16176 webserver.cc:460] Webserver started at http://127.0.0.1:45763/ using document root <none> and password file <none>
I20260812 11:29:14.845395 16176 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:14.845549 16176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:14.845857 16176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:14.849164 16176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "b3ce4087f1064c3b8e085868ff595707"
format_stamp: "Formatted at 2026-08-12 11:29:14 on dist-test-slave-h6n0"
server_key: "45564b3a719032d3d4f6da21d5d07065"
server_key_iv: "8f7aa26fbceeaf23d9e5c15e854ffd0b"
server_key_version: "encryptionkey@0"
I20260812 11:29:14.849977 16176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "b3ce4087f1064c3b8e085868ff595707"
format_stamp: "Formatted at 2026-08-12 11:29:14 on dist-test-slave-h6n0"
server_key: "45564b3a719032d3d4f6da21d5d07065"
server_key_iv: "8f7aa26fbceeaf23d9e5c15e854ffd0b"
server_key_version: "encryptionkey@0"
I20260812 11:29:14.856200 16176 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260812 11:29:14.859704 16201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:14.861037 16176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:29:14.861235 16176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "b3ce4087f1064c3b8e085868ff595707"
format_stamp: "Formatted at 2026-08-12 11:29:14 on dist-test-slave-h6n0"
server_key: "45564b3a719032d3d4f6da21d5d07065"
server_key_iv: "8f7aa26fbceeaf23d9e5c15e854ffd0b"
server_key_version: "encryptionkey@0"
I20260812 11:29:14.861469 16176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:14.879017 16176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:14.883670 16176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:14.884006 16176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:14.903533 16176 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45859
I20260812 11:29:14.903586 16283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45859 every 8 connection(s)
I20260812 11:29:14.904870 16176 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:14.908465 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16176
I20260812 11:29:14.908658 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:14.908679 16285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:14.909116 12731 external_mini_cluster.cc:1428] Setting key 6f7c61105bba18f9fedcf00bfffa5a4f
Aug 12 11:29:14 dist-test-slave-h6n0 krb5kdc[16149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:29:14.924873 16285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707: Bootstrap starting.
I20260812 11:29:14.928668 16285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:14.929803 16285 log.cc:824] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:14.932956 16285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707: No bootstrap required, opened a new log
I20260812 11:29:14.933668 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:14.910991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46357 (local address 127.0.0.1:45859)
0812 11:29:14.911812 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:14.911825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:14.911870 (+ 45us) server_negotiation.cc:407] Connection header received
0812 11:29:14.913788 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:14.913810 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:14.914177 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:14.914577 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:14.916165 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:14.917606 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:14.919012 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:14.919688 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:14.919889 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:14.925481 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:14.925490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:14.925504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:14.925544 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:14.929016 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:14.929827 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:14.929835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:14.929840 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:14.930362 (+ 522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:14.930876 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:14.930880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:14.930882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:14.931428 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:14.931748 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:14.932226 (+ 478us) server_negotiation.cc:299] Negotiation successful
0812 11:29:14.932633 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":403,"threads_started":1}
I20260812 11:29:14.937327 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:14.941897 16285 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ce4087f1064c3b8e085868ff595707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 11:29:14.942350 16285 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:14.942467 16285 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3ce4087f1064c3b8e085868ff595707, State: Initialized, Role: FOLLOWER
I20260812 11:29:14.943267 16285 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ce4087f1064c3b8e085868ff595707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 11:29:14.943609 16285 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:14.943773 16285 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:14.943984 16285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:14.947386 16285 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ce4087f1064c3b8e085868ff595707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 11:29:14.948031 16285 leader_election.cc:302] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3ce4087f1064c3b8e085868ff595707; no voters:
I20260812 11:29:14.948820 16285 leader_election.cc:288] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:14.948917 16299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:14.949748 16299 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [term 1 LEADER]: Becoming Leader. State: Replica: b3ce4087f1064c3b8e085868ff595707, State: Running, Role: LEADER
I20260812 11:29:14.950503 16299 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ce4087f1064c3b8e085868ff595707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260812 11:29:14.951354 16285 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:14.954393 16300 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3ce4087f1064c3b8e085868ff595707. Latest consensus state: current_term: 1 leader_uuid: "b3ce4087f1064c3b8e085868ff595707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ce4087f1064c3b8e085868ff595707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } } }
I20260812 11:29:14.954913 16300 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:14.959280 16301 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3ce4087f1064c3b8e085868ff595707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3ce4087f1064c3b8e085868ff595707" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } } }
I20260812 11:29:14.960300 16301 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:14.962280 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:14.941802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46607 (local address 127.0.0.1:45859)
0812 11:29:14.942010 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:14.942016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:14.942489 (+ 473us) server_negotiation.cc:407] Connection header received
0812 11:29:14.942709 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:14.942722 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:14.942781 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:14.942894 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:14.945105 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:14.945864 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:14.947932 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:14.948521 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:14.948665 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:14.951919 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:14.951925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:14.951930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:14.951965 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:14.956387 (+ 4422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:14.957522 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:14.957531 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:14.957536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:14.957718 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:14.958308 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:14.958315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:14.958321 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:14.958856 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:14.961425 (+ 2569us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:14.961615 (+ 190us) server_negotiation.cc:299] Negotiation successful
0812 11:29:14.961873 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:29:14.965361 16307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:14.970518 16307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:14.979221 16307 catalog_manager.cc:1347] Generated new cluster ID: 0781a52eee1b476a8727d4165ed80141
I20260812 11:29:14.979353 16307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:14.993722 16307 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:14.995575 16307 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:15.010059 16307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3ce4087f1064c3b8e085868ff595707: Generated new TSK 0
I20260812 11:29:15.011147 16307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:15.020203 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16176
I20260812 11:29:15.021733 16278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:15.112702 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16176
2026-08-12T11:29:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2403 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:15 dist-test-slave-h6n0 krb5kdc[16337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:15 dist-test-slave-h6n0 krb5kdc[16337](info): set up 2 sockets
Aug 12 11:29:15 dist-test-slave-h6n0 krb5kdc[16337](info): commencing operation
krb5kdc: starting...
W20260812 11:29:16.541846 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.383s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:16 dist-test-slave-h6n0 krb5kdc[16337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534156, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:16.905973 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:17.273100 16362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:17.273427 16362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:17.273533 16362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:17.281523 16362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:17.281658 16362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:17.281711 16362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:17.281781 16362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:17.281847 16362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:17.292776 16362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:17.294109 16362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:17.295480 16362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:17.308300 16369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:17.308365 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:17.308348 16368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:17.309198 16362 server_base.cc:1034] running on GCE node
I20260812 11:29:17.310082 16362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:17.311671 16362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:17.312942 16362 hybrid_clock.cc:648] HybridClock initialized: now 1786534157312876 us; error 45 us; skew 500 ppm
I20260812 11:29:17.313352 16362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:17 dist-test-slave-h6n0 krb5kdc[16337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534157, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:17.317106 16362 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:17.318970 16362 webserver.cc:460] Webserver started at http://127.0.0.1:33667/ using document root <none> and password file <none>
I20260812 11:29:17.319720 16362 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:17.319900 16362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:17.320242 16362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:17.324882 16362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "0afbe6c31a9240babb244645c8dab6ff"
format_stamp: "Formatted at 2026-08-12 11:29:17 on dist-test-slave-h6n0"
server_key: "6e344ba6092a4fcae2b31542d5dba7a2"
server_key_iv: "1293bd8adabd536e1a79106d0f78bcf6"
server_key_version: "encryptionkey@0"
I20260812 11:29:17.325855 16362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "0afbe6c31a9240babb244645c8dab6ff"
format_stamp: "Formatted at 2026-08-12 11:29:17 on dist-test-slave-h6n0"
server_key: "6e344ba6092a4fcae2b31542d5dba7a2"
server_key_iv: "1293bd8adabd536e1a79106d0f78bcf6"
server_key_version: "encryptionkey@0"
I20260812 11:29:17.332448 16362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260812 11:29:17.336421 16387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:17.337785 16362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:29:17.338003 16362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "0afbe6c31a9240babb244645c8dab6ff"
format_stamp: "Formatted at 2026-08-12 11:29:17 on dist-test-slave-h6n0"
server_key: "6e344ba6092a4fcae2b31542d5dba7a2"
server_key_iv: "1293bd8adabd536e1a79106d0f78bcf6"
server_key_version: "encryptionkey@0"
I20260812 11:29:17.338214 16362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:17.359418 16362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:17.363690 16362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:17.364018 16362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:17.386610 16362 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45485
I20260812 11:29:17.386633 16460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45485 every 8 connection(s)
I20260812 11:29:17.388073 16362 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:17.390296 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16362
I20260812 11:29:17.390497 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:17.391124 12731 external_mini_cluster.cc:1428] Setting key 441e618c230065e0c8993f68fff18d88
I20260812 11:29:17.393118 16461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:17.406391 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:17.392932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55847 (local address 127.0.0.1:45485)
0812 11:29:17.393731 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:17.393746 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:17.393794 (+ 48us) server_negotiation.cc:407] Connection header received
0812 11:29:17.396160 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:17.396187 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:17.396682 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:17.397116 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:17.397614 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:17.399667 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:17.401110 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:17.402158 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:17.402342 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:17.403617 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:17.403632 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:17.403648 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:17.404389 (+ 741us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:17.404483 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:17.404525 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:17.404634 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:17.404872 (+ 238us) server_negotiation.cc:299] Negotiation successful
0812 11:29:17.405141 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":346,"threads_started":1}
I20260812 11:29:17.407567 16461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff: Bootstrap starting.
I20260812 11:29:17.410218 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:17.410890 16461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:17.412110 16461 log.cc:824] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:17.415407 16461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff: No bootstrap required, opened a new log
I20260812 11:29:17.422039 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:17.415384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59337 (local address 127.0.0.1:45485)
0812 11:29:17.415569 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:17.415574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:17.416104 (+ 530us) server_negotiation.cc:407] Connection header received
0812 11:29:17.416288 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:17.416293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:17.416363 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:17.416459 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:17.416962 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:17.418103 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:17.419418 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:17.420069 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:17.420210 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:17.421258 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:17.421265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:17.421270 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:17.421289 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:17.421340 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:17.421361 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:17.421458 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:17.421576 (+ 118us) server_negotiation.cc:299] Negotiation successful
0812 11:29:17.421670 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:29:17.425014 16461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbe6c31a9240babb244645c8dab6ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260812 11:29:17.425354 16461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:17.425427 16461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afbe6c31a9240babb244645c8dab6ff, State: Initialized, Role: FOLLOWER
I20260812 11:29:17.425907 16461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbe6c31a9240babb244645c8dab6ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260812 11:29:17.426113 16461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:17.426211 16461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:17.426343 16461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:17.428659 16461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbe6c31a9240babb244645c8dab6ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260812 11:29:17.429292 16461 leader_election.cc:302] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afbe6c31a9240babb244645c8dab6ff; no voters:
I20260812 11:29:17.429872 16461 leader_election.cc:288] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:17.430032 16478 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:17.430773 16478 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [term 1 LEADER]: Becoming Leader. State: Replica: 0afbe6c31a9240babb244645c8dab6ff, State: Running, Role: LEADER
I20260812 11:29:17.431346 16478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbe6c31a9240babb244645c8dab6ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } }
I20260812 11:29:17.432066 16461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:17.433923 16480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0afbe6c31a9240babb244645c8dab6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbe6c31a9240babb244645c8dab6ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } } }
I20260812 11:29:17.434280 16480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:17.435246 16481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afbe6c31a9240babb244645c8dab6ff. Latest consensus state: current_term: 1 leader_uuid: "0afbe6c31a9240babb244645c8dab6ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afbe6c31a9240babb244645c8dab6ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45485 } } }
I20260812 11:29:17.435724 16481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:17.438737 16492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:17.444515 16492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:17.453644 16492 catalog_manager.cc:1347] Generated new cluster ID: 8484ce4a9937473a8c0963c83677c3b8
I20260812 11:29:17.453752 16492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:17.471083 16492 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:17.472391 16492 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:17.487977 16492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0afbe6c31a9240babb244645c8dab6ff: Generated new TSK 0
I20260812 11:29:17.488847 16492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:17.495826 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16362
W20260812 11:29:17.497162 16454 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55847
W20260812 11:29:17.497947 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:17.498113 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16362
2026-08-12T11:29:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:17 dist-test-slave-h6n0 krb5kdc[16520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:17 dist-test-slave-h6n0 krb5kdc[16520](info): set up 2 sockets
Aug 12 11:29:17 dist-test-slave-h6n0 krb5kdc[16520](info): commencing operation
krb5kdc: starting...
W20260812 11:29:18.934496 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.389s user 0.002s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:19 dist-test-slave-h6n0 krb5kdc[16520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:19.295002 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:19.651710 16547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:19.652017 16547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:19.652151 16547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:19.659646 16547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:19.659768 16547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:19.659857 16547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:19.659951 16547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:19.660029 16547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:19.670454 16547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:19.671895 16547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:19.673264 16547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:19.685670 16556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:19.685772 16557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:19.685935 16560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:19.686875 16547 server_base.cc:1034] running on GCE node
I20260812 11:29:19.687685 16547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:19.689190 16547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:19.690508 16547 hybrid_clock.cc:648] HybridClock initialized: now 1786534159690476 us; error 56 us; skew 500 ppm
I20260812 11:29:19.690975 16547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:19 dist-test-slave-h6n0 krb5kdc[16520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534159, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:19.698722 16547 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:19.700455 16547 webserver.cc:460] Webserver started at http://127.0.0.1:35055/ using document root <none> and password file <none>
I20260812 11:29:19.701207 16547 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:19.701320 16547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:19.701663 16547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:19.704886 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "00120f26c081422d826d6ad5c87867f5"
format_stamp: "Formatted at 2026-08-12 11:29:19 on dist-test-slave-h6n0"
server_key: "87c807d3b36f7f8c57bd1ace4eb943ff"
server_key_iv: "46b7e127c8dc9a511807cdf2776ac3ee"
server_key_version: "encryptionkey@0"
I20260812 11:29:19.705708 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "00120f26c081422d826d6ad5c87867f5"
format_stamp: "Formatted at 2026-08-12 11:29:19 on dist-test-slave-h6n0"
server_key: "87c807d3b36f7f8c57bd1ace4eb943ff"
server_key_iv: "46b7e127c8dc9a511807cdf2776ac3ee"
server_key_version: "encryptionkey@0"
I20260812 11:29:19.711922 16547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260812 11:29:19.717031 16567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:19.718895 16547 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260812 11:29:19.719130 16547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "00120f26c081422d826d6ad5c87867f5"
format_stamp: "Formatted at 2026-08-12 11:29:19 on dist-test-slave-h6n0"
server_key: "87c807d3b36f7f8c57bd1ace4eb943ff"
server_key_iv: "46b7e127c8dc9a511807cdf2776ac3ee"
server_key_version: "encryptionkey@0"
I20260812 11:29:19.719373 16547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:19.726169 16547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:19.731993 16547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:19.732411 16547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:19.756330 16547 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46005
I20260812 11:29:19.756451 16659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46005 every 8 connection(s)
I20260812 11:29:19.757958 16547 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:19.759327 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16547
I20260812 11:29:19.759539 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:19.760102 12731 external_mini_cluster.cc:1428] Setting key ade22df9994555a67d9730e4649369d5
I20260812 11:29:19.762817 16660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:19.769642 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:19.762059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41765 (local address 127.0.0.1:46005)
0812 11:29:19.763030 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:19.763043 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:19.763093 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:29:19.765034 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:19.765054 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:19.765416 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:19.765756 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:19.767064 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:19.767074 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:19.767093 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:19.767120 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:19.767297 (+ 177us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:19.767340 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:19.767612 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:19.767758 (+ 146us) server_negotiation.cc:299] Negotiation successful
0812 11:29:19.768154 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":444,"threads_started":1}
I20260812 11:29:19.774152 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:19.777271 16660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5: Bootstrap starting.
I20260812 11:29:19.781690 16660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:19.783224 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:19.780329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56841 (local address 127.0.0.1:46005)
0812 11:29:19.780567 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:19.780574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:19.780925 (+ 351us) server_negotiation.cc:407] Connection header received
0812 11:29:19.781143 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:19.781146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:19.781197 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:19.781282 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:19.782465 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:19.782469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:19.782473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:19.782487 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:19.782577 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:19.782605 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:19.782701 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:19.782859 (+ 158us) server_negotiation.cc:299] Negotiation successful
0812 11:29:19.782912 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260812 11:29:19.783270 16660 log.cc:824] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:19.786307 16660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5: No bootstrap required, opened a new log
I20260812 11:29:19.795142 16660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00120f26c081422d826d6ad5c87867f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46005 } }
I20260812 11:29:19.795729 16660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:19.795861 16660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00120f26c081422d826d6ad5c87867f5, State: Initialized, Role: FOLLOWER
I20260812 11:29:19.796480 16660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00120f26c081422d826d6ad5c87867f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46005 } }
I20260812 11:29:19.796759 16660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:19.796943 16660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:19.797155 16660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:19.799691 16660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00120f26c081422d826d6ad5c87867f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46005 } }
I20260812 11:29:19.800339 16660 leader_election.cc:302] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00120f26c081422d826d6ad5c87867f5; no voters:
I20260812 11:29:19.800994 16660 leader_election.cc:288] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:19.801111 16673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:19.801926 16673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [term 1 LEADER]: Becoming Leader. State: Replica: 00120f26c081422d826d6ad5c87867f5, State: Running, Role: LEADER
I20260812 11:29:19.802606 16673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00120f26c081422d826d6ad5c87867f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46005 } }
I20260812 11:29:19.803460 16660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:19.805274 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00120f26c081422d826d6ad5c87867f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00120f26c081422d826d6ad5c87867f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46005 } } }
I20260812 11:29:19.805680 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:19.806203 16678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00120f26c081422d826d6ad5c87867f5. Latest consensus state: current_term: 1 leader_uuid: "00120f26c081422d826d6ad5c87867f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00120f26c081422d826d6ad5c87867f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46005 } } }
I20260812 11:29:19.806660 16678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:19.808585 16683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:19.815621 16683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:19.825453 16683 catalog_manager.cc:1347] Generated new cluster ID: a38227f7d721404d9fa89754f7b652e4
I20260812 11:29:19.825595 16683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:19.843449 16683 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:19.844275 16683 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:19.850660 16683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00120f26c081422d826d6ad5c87867f5: Generated new TSK 0
I20260812 11:29:19.851660 16683 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:19.859869 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16547
W20260812 11:29:19.861078 16654 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41765
W20260812 11:29:19.861830 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16547: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:19.862108 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16547
2026-08-12T11:29:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:19 dist-test-slave-h6n0 krb5kdc[16709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:19 dist-test-slave-h6n0 krb5kdc[16709](info): set up 2 sockets
Aug 12 11:29:19 dist-test-slave-h6n0 krb5kdc[16709](info): commencing operation
krb5kdc: starting...
W20260812 11:29:21.275503 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.362s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:21 dist-test-slave-h6n0 krb5kdc[16709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:21.645429 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:22.008769 16727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:22.009145 16727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:22.009323 16727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:22.018292 16727 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:22.018473 16727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:22.018615 16727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:22.018741 16727 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:22.019016 16727 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:22.031428 16727 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:22.033133 16727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:22.034550 16727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:22.046499 16740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:22.046551 16744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:22.046617 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:22.047879 16727 server_base.cc:1034] running on GCE node
I20260812 11:29:22.048743 16727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:22.050421 16727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:22.051657 16727 hybrid_clock.cc:648] HybridClock initialized: now 1786534162051636 us; error 79 us; skew 500 ppm
I20260812 11:29:22.052085 16727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:22 dist-test-slave-h6n0 krb5kdc[16709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:22.055449 16727 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:22.057117 16727 webserver.cc:460] Webserver started at http://127.0.0.1:34087/ using document root <none> and password file <none>
I20260812 11:29:22.057833 16727 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:22.057999 16727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:22.058319 16727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:22.061416 16727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "b9aa69010cdb43e28fb7b639feab50dd"
format_stamp: "Formatted at 2026-08-12 11:29:22 on dist-test-slave-h6n0"
server_key: "ffb882c3951299540a7cb2c39a456911"
server_key_iv: "8ddf7072c262ba5f28ad279ee05efca8"
server_key_version: "encryptionkey@0"
I20260812 11:29:22.062319 16727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "b9aa69010cdb43e28fb7b639feab50dd"
format_stamp: "Formatted at 2026-08-12 11:29:22 on dist-test-slave-h6n0"
server_key: "ffb882c3951299540a7cb2c39a456911"
server_key_iv: "8ddf7072c262ba5f28ad279ee05efca8"
server_key_version: "encryptionkey@0"
I20260812 11:29:22.067657 16727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:29:22.072111 16752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:22.073827 16727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260812 11:29:22.074031 16727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "b9aa69010cdb43e28fb7b639feab50dd"
format_stamp: "Formatted at 2026-08-12 11:29:22 on dist-test-slave-h6n0"
server_key: "ffb882c3951299540a7cb2c39a456911"
server_key_iv: "8ddf7072c262ba5f28ad279ee05efca8"
server_key_version: "encryptionkey@0"
I20260812 11:29:22.074240 16727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:22.088550 16727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:22.092753 16727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:22.093042 16727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:22.112535 16727 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34113
I20260812 11:29:22.112579 16834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34113 every 8 connection(s)
I20260812 11:29:22.113973 16727 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:22.117584 16836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:22.123383 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16727
I20260812 11:29:22.123600 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:22.124042 12731 external_mini_cluster.cc:1428] Setting key d592a8e9bf38b37e205698e9b06f433b
I20260812 11:29:22.128053 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd: Bootstrap starting.
I20260812 11:29:22.131671 16836 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:22.132812 16836 log.cc:824] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:22.135783 16836 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd: No bootstrap required, opened a new log
Aug 12 11:29:22 dist-test-slave-h6n0 krb5kdc[16709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:29:22.145748 16836 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9aa69010cdb43e28fb7b639feab50dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260812 11:29:22.146126 16836 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:22.146229 16836 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9aa69010cdb43e28fb7b639feab50dd, State: Initialized, Role: FOLLOWER
I20260812 11:29:22.146938 16836 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9aa69010cdb43e28fb7b639feab50dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260812 11:29:22.147256 16836 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:22.147413 16836 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:22.147609 16836 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:22.149075 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:22.125891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:43187 (local address 127.0.0.1:34113)
0812 11:29:22.126721 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:22.126732 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:22.126770 (+ 38us) server_negotiation.cc:407] Connection header received
0812 11:29:22.128452 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:22.128472 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:22.128804 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:22.129103 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:22.131039 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:22.132364 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:22.134000 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:22.134702 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:22.134832 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:22.140034 (+ 5202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:22.140042 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:22.140067 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:22.140109 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:22.143975 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:22.144996 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:22.145002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:22.145008 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:22.145174 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:22.145725 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:22.145729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:22.145731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:22.146458 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:22.146848 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:22.147593 (+ 745us) server_negotiation.cc:299] Negotiation successful
0812 11:29:22.148020 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":368,"threads_started":1}
I20260812 11:29:22.150075 16836 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9aa69010cdb43e28fb7b639feab50dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260812 11:29:22.150703 16836 leader_election.cc:302] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9aa69010cdb43e28fb7b639feab50dd; no voters:
I20260812 11:29:22.151403 16836 leader_election.cc:288] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:22.151613 16846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:22.153373 16846 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [term 1 LEADER]: Becoming Leader. State: Replica: b9aa69010cdb43e28fb7b639feab50dd, State: Running, Role: LEADER
I20260812 11:29:22.153918 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:22.153945 16846 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9aa69010cdb43e28fb7b639feab50dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } }
I20260812 11:29:22.154567 16836 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:22.156978 16850 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9aa69010cdb43e28fb7b639feab50dd. Latest consensus state: current_term: 1 leader_uuid: "b9aa69010cdb43e28fb7b639feab50dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9aa69010cdb43e28fb7b639feab50dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } } }
I20260812 11:29:22.157438 16850 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:22.159994 16847 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9aa69010cdb43e28fb7b639feab50dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9aa69010cdb43e28fb7b639feab50dd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34113 } } }
I20260812 11:29:22.160351 16847 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:22.166990 16855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:22.172477 16855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:22.185520 16855 catalog_manager.cc:1347] Generated new cluster ID: 0ebb2a6b0ac943059398b3e8cf033286
I20260812 11:29:22.185642 16855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:22.196648 16855 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:22.197824 16855 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:22.200589 16844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:22.169905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:50809 (local address 127.0.0.1:34113)
0812 11:29:22.170100 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:22.170107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:22.182841 (+ 12734us) server_negotiation.cc:407] Connection header received
0812 11:29:22.183083 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:22.183095 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:22.183213 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:22.183373 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:22.185544 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:22.186819 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:22.188580 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:22.189257 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:22.189397 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:22.192427 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:22.192435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:22.192440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:22.192478 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:22.196413 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:22.197398 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:22.197404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:22.197408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:22.197567 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:22.198146 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:22.198151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:22.198154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:22.198574 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:22.198861 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:22.200010 (+ 1149us) server_negotiation.cc:299] Negotiation successful
0812 11:29:22.200196 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:29:22.206645 16855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9aa69010cdb43e28fb7b639feab50dd: Generated new TSK 0
I20260812 11:29:22.207417 16855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:22.212880 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16727
I20260812 11:29:22.214972 16823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:22.312036 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16727
W20260812 11:29:22.339597 16706 connection.cc:535] client connection to 127.0.0.1:34113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34113): Connection reset by peer (error 104)
2026-08-12T11:29:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:22 dist-test-slave-h6n0 krb5kdc[16887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:22 dist-test-slave-h6n0 krb5kdc[16887](info): set up 2 sockets
Aug 12 11:29:22 dist-test-slave-h6n0 krb5kdc[16887](info): commencing operation
krb5kdc: starting...
W20260812 11:29:23.734427 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.376s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:23 dist-test-slave-h6n0 krb5kdc[16887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:24.076000 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:24.419346 16905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:24.419646 16905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:24.419811 16905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:24.427608 16905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:24.427752 16905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:24.427845 16905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:24.427960 16905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:24.428090 16905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:24.437914 16905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:24.439210 16905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:24.440522 16905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:24.452481 16914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:24.452462 16921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:24.452512 16916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:24.453715 16905 server_base.cc:1034] running on GCE node
I20260812 11:29:24.454581 16905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:24.455929 16905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:24.457159 16905 hybrid_clock.cc:648] HybridClock initialized: now 1786534164457118 us; error 35 us; skew 500 ppm
I20260812 11:29:24.457500 16905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:24 dist-test-slave-h6n0 krb5kdc[16887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:24.464963 16905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:24.466626 16905 webserver.cc:460] Webserver started at http://127.0.0.1:43711/ using document root <none> and password file <none>
I20260812 11:29:24.467322 16905 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:24.467471 16905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:24.467828 16905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:24.471030 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "a8ed4afca6114d09b5188230e28c6a50"
format_stamp: "Formatted at 2026-08-12 11:29:24 on dist-test-slave-h6n0"
server_key: "db5e21b7b5c1ebe388de41335295a42e"
server_key_iv: "d0f28400172d34103470b6819c2d7a51"
server_key_version: "encryptionkey@0"
I20260812 11:29:24.472101 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "a8ed4afca6114d09b5188230e28c6a50"
format_stamp: "Formatted at 2026-08-12 11:29:24 on dist-test-slave-h6n0"
server_key: "db5e21b7b5c1ebe388de41335295a42e"
server_key_iv: "d0f28400172d34103470b6819c2d7a51"
server_key_version: "encryptionkey@0"
I20260812 11:29:24.477144 16905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260812 11:29:24.480561 16929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:24.481841 16905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:29:24.482069 16905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "a8ed4afca6114d09b5188230e28c6a50"
format_stamp: "Formatted at 2026-08-12 11:29:24 on dist-test-slave-h6n0"
server_key: "db5e21b7b5c1ebe388de41335295a42e"
server_key_iv: "d0f28400172d34103470b6819c2d7a51"
server_key_version: "encryptionkey@0"
I20260812 11:29:24.482281 16905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:24.502712 16905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:24.507609 16905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:24.507952 16905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:24.531203 16905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42063
I20260812 11:29:24.531237 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42063 every 8 connection(s)
I20260812 11:29:24.532711 16905 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:24.536460 17010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:24.539163 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 16905
I20260812 11:29:24.539371 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:24.539964 12731 external_mini_cluster.cc:1428] Setting key f1740b9d9febc1c9a2f46b1978bf8e04
I20260812 11:29:24.550981 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50: Bootstrap starting.
I20260812 11:29:24.554816 17010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:24.555101 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:24.541932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:47791 (local address 127.0.0.1:42063)
0812 11:29:24.542850 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:24.542861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:24.542905 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:29:24.544676 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:24.544697 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:24.545077 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:24.545400 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:24.546436 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:24.547906 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:24.550167 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:24.550971 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:24.551169 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:24.552540 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:24.552548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:24.552559 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:24.553132 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:24.553219 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:24.553246 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:24.553418 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:24.553830 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 11:29:24.554025 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":410,"threads_started":1}
I20260812 11:29:24.556066 17010 log.cc:824] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:24.559028 17010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50: No bootstrap required, opened a new log
I20260812 11:29:24.559226 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:24.567636 17010 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed4afca6114d09b5188230e28c6a50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260812 11:29:24.568100 17010 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:24.568267 17010 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ed4afca6114d09b5188230e28c6a50, State: Initialized, Role: FOLLOWER
I20260812 11:29:24.568842 17010 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed4afca6114d09b5188230e28c6a50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260812 11:29:24.569144 17010 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:24.569316 17010 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:24.569500 17010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:24.571183 17013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:24.563698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:54897 (local address 127.0.0.1:42063)
0812 11:29:24.563938 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:24.563943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:24.564289 (+ 346us) server_negotiation.cc:407] Connection header received
0812 11:29:24.564463 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:24.564470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:24.564533 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:24.564626 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:24.565267 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:24.566284 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:24.568253 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:24.568915 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:24.569025 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:24.570300 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:24.570305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:24.570308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:24.570335 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:24.570376 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:24.570391 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:24.570487 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:24.570760 (+ 273us) server_negotiation.cc:299] Negotiation successful
0812 11:29:24.570823 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:29:24.572710 17010 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed4afca6114d09b5188230e28c6a50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260812 11:29:24.573249 17010 leader_election.cc:302] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8ed4afca6114d09b5188230e28c6a50; no voters:
I20260812 11:29:24.573832 17010 leader_election.cc:288] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:24.574044 17026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:24.574913 17026 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [term 1 LEADER]: Becoming Leader. State: Replica: a8ed4afca6114d09b5188230e28c6a50, State: Running, Role: LEADER
I20260812 11:29:24.575611 17026 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed4afca6114d09b5188230e28c6a50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } }
I20260812 11:29:24.576118 17010 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:24.579201 17027 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8ed4afca6114d09b5188230e28c6a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed4afca6114d09b5188230e28c6a50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } } }
I20260812 11:29:24.579556 17028 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8ed4afca6114d09b5188230e28c6a50. Latest consensus state: current_term: 1 leader_uuid: "a8ed4afca6114d09b5188230e28c6a50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ed4afca6114d09b5188230e28c6a50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42063 } } }
I20260812 11:29:24.579968 17027 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:24.579973 17028 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:24.585258 17036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:24.591459 17036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:24.601181 17036 catalog_manager.cc:1347] Generated new cluster ID: 5f79b25c90a44b8886940eb49ae712d1
I20260812 11:29:24.601318 17036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:24.610453 17036 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:24.611943 17036 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:24.625092 17036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8ed4afca6114d09b5188230e28c6a50: Generated new TSK 0
I20260812 11:29:24.625816 17036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:24.639855 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16905
W20260812 11:29:24.641742 17000 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:47791
W20260812 11:29:24.642724 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 16905: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:24.642899 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 16905
W20260812 11:29:24.679503 16885 connection.cc:535] client connection to 127.0.0.1:42063 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42063): Connection reset by peer (error 104)
2026-08-12T11:29:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:24 dist-test-slave-h6n0 krb5kdc[17053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:24 dist-test-slave-h6n0 krb5kdc[17053](info): set up 2 sockets
Aug 12 11:29:24 dist-test-slave-h6n0 krb5kdc[17053](info): commencing operation
krb5kdc: starting...
W20260812 11:29:26.087188 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.386s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:26 dist-test-slave-h6n0 krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534166, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:26.435838 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:26.803468 17071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:26.803846 17071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:26.804031 17071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:26.812961 17071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:26.813141 17071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:26.813263 17071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:26.813397 17071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:26.813486 17071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:26.823693 17071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:26.825047 17071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:26.826416 17071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:26.838775 17077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:26.838704 17078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:26.838737 17084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:26.840122 17071 server_base.cc:1034] running on GCE node
I20260812 11:29:26.840970 17071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:26.842495 17071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:26.843797 17071 hybrid_clock.cc:648] HybridClock initialized: now 1786534166843754 us; error 36 us; skew 500 ppm
I20260812 11:29:26.844143 17071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:26 dist-test-slave-h6n0 krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:26.847386 17071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:26.848977 17071 webserver.cc:460] Webserver started at http://127.0.0.1:34601/ using document root <none> and password file <none>
I20260812 11:29:26.850106 17071 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:26.850592 17071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:26.851152 17071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:26.855053 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "40ddd09ddadf4e1f876385ab8e76147c"
format_stamp: "Formatted at 2026-08-12 11:29:26 on dist-test-slave-h6n0"
server_key: "d4599589abf88bdf06214f4213b8a5ce"
server_key_iv: "8c4c29fa9de6061a73a75b40a7c0cf3c"
server_key_version: "encryptionkey@0"
I20260812 11:29:26.855950 17071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "40ddd09ddadf4e1f876385ab8e76147c"
format_stamp: "Formatted at 2026-08-12 11:29:26 on dist-test-slave-h6n0"
server_key: "d4599589abf88bdf06214f4213b8a5ce"
server_key_iv: "8c4c29fa9de6061a73a75b40a7c0cf3c"
server_key_version: "encryptionkey@0"
I20260812 11:29:26.861272 17071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260812 11:29:26.864781 17095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:26.866107 17071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:29:26.866302 17071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "40ddd09ddadf4e1f876385ab8e76147c"
format_stamp: "Formatted at 2026-08-12 11:29:26 on dist-test-slave-h6n0"
server_key: "d4599589abf88bdf06214f4213b8a5ce"
server_key_iv: "8c4c29fa9de6061a73a75b40a7c0cf3c"
server_key_version: "encryptionkey@0"
I20260812 11:29:26.866506 17071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:26.872301 17071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:26.876251 17071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:26.876562 17071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:26.895910 17071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36175
I20260812 11:29:26.896035 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36175 every 8 connection(s)
I20260812 11:29:26.898056 17071 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:26.900557 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17071
I20260812 11:29:26.900761 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:26.901363 12731 external_mini_cluster.cc:1428] Setting key fe73bfa381d2a1f52c0b656839928fe4
I20260812 11:29:26.902948 17172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:26.909428 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:26.903543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:43207 (local address 127.0.0.1:36175)
0812 11:29:26.904374 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:26.904386 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:26.904431 (+ 45us) server_negotiation.cc:407] Connection header received
0812 11:29:26.906029 (+ 1598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:26.906050 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:26.906383 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:26.906759 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:26.907838 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:26.907843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:26.907855 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:26.907869 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:26.907960 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:26.907980 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:26.908161 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:26.908298 (+ 137us) server_negotiation.cc:299] Negotiation successful
0812 11:29:26.908521 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":399,"threads_started":1}
I20260812 11:29:26.913938 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:26.916378 17172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c: Bootstrap starting.
I20260812 11:29:26.920043 17172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:26.921360 17178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:26.918581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:51397 (local address 127.0.0.1:36175)
0812 11:29:26.918780 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:26.918785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:26.919122 (+ 337us) server_negotiation.cc:407] Connection header received
0812 11:29:26.919406 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:26.919411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:26.919462 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:26.919572 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:26.920676 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:26.920680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:26.920683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:26.920698 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:26.920737 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:26.920752 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:26.920849 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:26.920997 (+ 148us) server_negotiation.cc:299] Negotiation successful
0812 11:29:26.921045 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260812 11:29:26.921556 17172 log.cc:824] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:26.924853 17172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c: No bootstrap required, opened a new log
I20260812 11:29:26.932878 17172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddd09ddadf4e1f876385ab8e76147c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36175 } }
I20260812 11:29:26.933305 17172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:26.933431 17172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ddd09ddadf4e1f876385ab8e76147c, State: Initialized, Role: FOLLOWER
I20260812 11:29:26.934031 17172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddd09ddadf4e1f876385ab8e76147c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36175 } }
I20260812 11:29:26.934299 17172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:26.934476 17172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:26.934707 17172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:26.937278 17172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddd09ddadf4e1f876385ab8e76147c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36175 } }
I20260812 11:29:26.937829 17172 leader_election.cc:302] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ddd09ddadf4e1f876385ab8e76147c; no voters:
I20260812 11:29:26.938444 17172 leader_election.cc:288] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:26.938628 17186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:26.939533 17186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [term 1 LEADER]: Becoming Leader. State: Replica: 40ddd09ddadf4e1f876385ab8e76147c, State: Running, Role: LEADER
I20260812 11:29:26.940092 17186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddd09ddadf4e1f876385ab8e76147c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36175 } }
I20260812 11:29:26.940907 17172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:26.942893 17188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40ddd09ddadf4e1f876385ab8e76147c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddd09ddadf4e1f876385ab8e76147c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36175 } } }
I20260812 11:29:26.944531 17188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:26.944185 17192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ddd09ddadf4e1f876385ab8e76147c. Latest consensus state: current_term: 1 leader_uuid: "40ddd09ddadf4e1f876385ab8e76147c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ddd09ddadf4e1f876385ab8e76147c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36175 } } }
I20260812 11:29:26.945115 17192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:26.951920 17205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:26.956992 17205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:26.965663 17205 catalog_manager.cc:1347] Generated new cluster ID: 49b4eafa9b104298aaca2e447ef14046
I20260812 11:29:26.965783 17205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:26.987095 17205 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:26.987589 17205 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:26.996699 17205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ddd09ddadf4e1f876385ab8e76147c: Generated new TSK 0
I20260812 11:29:26.997740 17205 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:27.006780 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17071
W20260812 11:29:27.008280 17161 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:43207
W20260812 11:29:27.009114 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17071: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:27.009287 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17071
2026-08-12T11:29:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:27 dist-test-slave-h6n0 krb5kdc[17223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:27 dist-test-slave-h6n0 krb5kdc[17223](info): set up 2 sockets
Aug 12 11:29:27 dist-test-slave-h6n0 krb5kdc[17223](info): commencing operation
krb5kdc: starting...
W20260812 11:29:28.472278 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.410s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:28 dist-test-slave-h6n0 krb5kdc[17223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:28.862635 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36143
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:29.214124 17246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:29.214404 17246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:29.214510 17246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:29.221405 17246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:29.221516 17246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:29.221562 17246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:29.221623 17246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:29.221668 17246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:29.230950 17246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36143
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:29.232041 17246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:29.233244 17246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:29.244601 17262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:29.244570 17261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:29.245158 17268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:29.247015 17246 server_base.cc:1034] running on GCE node
I20260812 11:29:29.247824 17246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:29.249341 17246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:29.250644 17246 hybrid_clock.cc:648] HybridClock initialized: now 1786534169250583 us; error 47 us; skew 500 ppm
I20260812 11:29:29.251034 17246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:29 dist-test-slave-h6n0 krb5kdc[17223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534169, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:29.254236 17246 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:29.255828 17246 webserver.cc:460] Webserver started at http://127.0.0.1:46179/ using document root <none> and password file <none>
I20260812 11:29:29.256479 17246 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:29.256585 17246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:29.256822 17246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:29.259815 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "261ddf75a9684702900a955b9d5738e1"
format_stamp: "Formatted at 2026-08-12 11:29:29 on dist-test-slave-h6n0"
server_key: "a4f5b5b3c7678df534d815896dbd8e54"
server_key_iv: "c7558cce5e613066aa740fd3fefa4adc"
server_key_version: "encryptionkey@0"
I20260812 11:29:29.260550 17246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "261ddf75a9684702900a955b9d5738e1"
format_stamp: "Formatted at 2026-08-12 11:29:29 on dist-test-slave-h6n0"
server_key: "a4f5b5b3c7678df534d815896dbd8e54"
server_key_iv: "c7558cce5e613066aa740fd3fefa4adc"
server_key_version: "encryptionkey@0"
I20260812 11:29:29.265563 17246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:29:29.268915 17282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:29.270165 17246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:29:29.270372 17246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "261ddf75a9684702900a955b9d5738e1"
format_stamp: "Formatted at 2026-08-12 11:29:29 on dist-test-slave-h6n0"
server_key: "a4f5b5b3c7678df534d815896dbd8e54"
server_key_iv: "c7558cce5e613066aa740fd3fefa4adc"
server_key_version: "encryptionkey@0"
I20260812 11:29:29.270591 17246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:29.290973 17246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:29.295396 17246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:29.295697 17246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:29.315510 17246 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36143
I20260812 11:29:29.315559 17373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36143 every 8 connection(s)
I20260812 11:29:29.316898 17246 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:29.320587 17374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:29.325316 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17246
I20260812 11:29:29.325512 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:29.326093 12731 external_mini_cluster.cc:1428] Setting key 8edf9f99ed4da7df1ef23fa34797a47e
I20260812 11:29:29.332252 17374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1: Bootstrap starting.
I20260812 11:29:29.335918 17374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:29.337069 17374 log.cc:824] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:29.340503 17374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1: No bootstrap required, opened a new log
Aug 12 11:29:29 dist-test-slave-h6n0 krb5kdc[17223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260812 11:29:29.348712 17374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ddf75a9684702900a955b9d5738e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260812 11:29:29.349097 17374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:29.349237 17374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 261ddf75a9684702900a955b9d5738e1, State: Initialized, Role: FOLLOWER
I20260812 11:29:29.349866 17374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ddf75a9684702900a955b9d5738e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260812 11:29:29.350138 17374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:29.350296 17374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:29.350498 17374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:29.350839 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:29.327898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:55407 (local address 127.0.0.1:36143)
0812 11:29:29.328718 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:29.328729 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:29.328769 (+ 40us) server_negotiation.cc:407] Connection header received
0812 11:29:29.330745 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:29.330767 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:29.331145 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:29.331450 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:29.333356 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:29.334796 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:29.336623 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:29.337388 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:29.337548 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:29.342624 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:29.342633 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:29.342650 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:29.342694 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:29.345936 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:29.346831 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:29.346841 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:29.346847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:29.347029 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:29.347518 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:29.347530 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:29.347532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:29.348072 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:29.348408 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:29.348938 (+ 530us) server_negotiation.cc:299] Negotiation successful
0812 11:29:29.349557 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":402,"threads_started":1}
I20260812 11:29:29.353510 17374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ddf75a9684702900a955b9d5738e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260812 11:29:29.354033 17374 leader_election.cc:302] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 261ddf75a9684702900a955b9d5738e1; no voters:
I20260812 11:29:29.354739 17374 leader_election.cc:288] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:29.354880 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:29.354858 17389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:29.357161 17374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:29.361721 17389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [term 1 LEADER]: Becoming Leader. State: Replica: 261ddf75a9684702900a955b9d5738e1, State: Running, Role: LEADER
I20260812 11:29:29.362648 17389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ddf75a9684702900a955b9d5738e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } }
I20260812 11:29:29.366969 17392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 261ddf75a9684702900a955b9d5738e1. Latest consensus state: current_term: 1 leader_uuid: "261ddf75a9684702900a955b9d5738e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ddf75a9684702900a955b9d5738e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260812 11:29:29.367801 17392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:29.367468 17391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "261ddf75a9684702900a955b9d5738e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "261ddf75a9684702900a955b9d5738e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36143 } } }
I20260812 11:29:29.368306 17391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1 [sys.catalog]: This master's current role is: LEADER
W20260812 11:29:29.373030 17419 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:29:29.373210 17419 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:29:29.375452 17422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:29.377744 17422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:29.381471 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:29.359772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:50909 (local address 127.0.0.1:36143)
0812 11:29:29.359904 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:29.359909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:29.362932 (+ 3023us) server_negotiation.cc:407] Connection header received
0812 11:29:29.363116 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:29.363125 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:29.363229 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:29.363351 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:29.365283 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:29.366651 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:29.370037 (+ 3386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:29.371115 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:29.371283 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:29.375326 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:29.375331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:29.375334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:29.375368 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:29.378437 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:29.379211 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:29.379215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:29.379217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:29.379332 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:29.379811 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:29.379815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:29.379817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:29.380147 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:29.380445 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:29.380868 (+ 423us) server_negotiation.cc:299] Negotiation successful
0812 11:29:29.381089 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260812 11:29:29.386322 17422 catalog_manager.cc:1347] Generated new cluster ID: da03797d3fbf4987a805cad8edd7d2a0
I20260812 11:29:29.386446 17422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:29.397012 17422 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:29.398711 17422 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:29.409818 17422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 261ddf75a9684702900a955b9d5738e1: Generated new TSK 0
I20260812 11:29:29.410750 17422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:29.416790 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17246
I20260812 11:29:29.417889 17367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:29.510663 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17246
W20260812 11:29:29.538851 17221 connection.cc:535] client connection to 127.0.0.1:36143 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36143): Connection reset by peer (error 104)
2026-08-12T11:29:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:29 dist-test-slave-h6n0 krb5kdc[17432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:29 dist-test-slave-h6n0 krb5kdc[17432](info): set up 2 sockets
Aug 12 11:29:29 dist-test-slave-h6n0 krb5kdc[17432](info): commencing operation
krb5kdc: starting...
W20260812 11:29:30.915503 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.355s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:31 dist-test-slave-h6n0 krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:31.254169 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:31.614133 17454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:31.614710 17454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:31.614946 17454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:31.622928 17454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:31.623073 17454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:31.623242 17454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:31.623345 17454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:31.623389 17454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:31.633647 17454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:31.635812 17454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:31.637841 17454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:31.652905 17459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:31.652995 17467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:31.653216 17462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:31.655045 17454 server_base.cc:1034] running on GCE node
I20260812 11:29:31.656183 17454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:31.657877 17454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:31.659101 17454 hybrid_clock.cc:648] HybridClock initialized: now 1786534171659063 us; error 117 us; skew 500 ppm
I20260812 11:29:31.659497 17454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:31 dist-test-slave-h6n0 krb5kdc[17432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:31.667131 17454 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:31.669214 17454 webserver.cc:460] Webserver started at http://127.0.0.1:46641/ using document root <none> and password file <none>
I20260812 11:29:31.669952 17454 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:31.670082 17454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:31.670384 17454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:31.674346 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "c1c51c922d5747c3ac607ca88c5f5e25"
format_stamp: "Formatted at 2026-08-12 11:29:31 on dist-test-slave-h6n0"
server_key: "1f87862b9dfa4893ccb6302f5f5428ef"
server_key_iv: "775eb89bd6cc6e54fc571dcac357b79a"
server_key_version: "encryptionkey@0"
I20260812 11:29:31.675451 17454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "c1c51c922d5747c3ac607ca88c5f5e25"
format_stamp: "Formatted at 2026-08-12 11:29:31 on dist-test-slave-h6n0"
server_key: "1f87862b9dfa4893ccb6302f5f5428ef"
server_key_iv: "775eb89bd6cc6e54fc571dcac357b79a"
server_key_version: "encryptionkey@0"
I20260812 11:29:31.681631 17454 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260812 11:29:31.685298 17476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:31.686770 17454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 11:29:31.686967 17454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "c1c51c922d5747c3ac607ca88c5f5e25"
format_stamp: "Formatted at 2026-08-12 11:29:31 on dist-test-slave-h6n0"
server_key: "1f87862b9dfa4893ccb6302f5f5428ef"
server_key_iv: "775eb89bd6cc6e54fc571dcac357b79a"
server_key_version: "encryptionkey@0"
I20260812 11:29:31.687191 17454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:31.703354 17454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:31.707906 17454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:31.708266 17454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:31.730326 17454 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41965
I20260812 11:29:31.730351 17563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41965 every 8 connection(s)
I20260812 11:29:31.731880 17454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:31.735642 17564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:31.737851 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17454
I20260812 11:29:31.738052 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:31.738632 12731 external_mini_cluster.cc:1428] Setting key 35adac01b7d062b9e69c1a05757e02c5
I20260812 11:29:31.747987 17564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25: Bootstrap starting.
I20260812 11:29:31.751873 17564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:31.752903 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:31.740432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:35593 (local address 127.0.0.1:41965)
0812 11:29:31.741289 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:31.741303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:31.741353 (+ 50us) server_negotiation.cc:407] Connection header received
0812 11:29:31.743126 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:31.743148 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:31.743514 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:31.743821 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:31.744436 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:31.745761 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:31.747615 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:31.748353 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:31.748488 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:31.749904 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:31.749912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:31.749924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:31.750469 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:31.750616 (+ 147us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:31.750648 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:31.750784 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:31.751472 (+ 688us) server_negotiation.cc:299] Negotiation successful
0812 11:29:31.751815 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":389,"threads_started":1}
I20260812 11:29:31.753000 17564 log.cc:824] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:31.756351 17564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25: No bootstrap required, opened a new log
I20260812 11:29:31.757227 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:31.765631 17564 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260812 11:29:31.766125 17564 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:31.766252 17564 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1c51c922d5747c3ac607ca88c5f5e25, State: Initialized, Role: FOLLOWER
I20260812 11:29:31.766944 17564 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260812 11:29:31.767295 17564 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:31.767478 17564 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:31.767666 17564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:31.770041 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:31.761935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:39269 (local address 127.0.0.1:41965)
0812 11:29:31.762146 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:31.762153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:31.762625 (+ 472us) server_negotiation.cc:407] Connection header received
0812 11:29:31.762833 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:31.762837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:31.762909 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:31.763022 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:31.764067 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:31.765052 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:31.767248 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:31.767917 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:31.768029 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:31.769238 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:31.769242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:31.769246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:31.769263 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:31.769301 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:31.769319 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:31.769409 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:31.769638 (+ 229us) server_negotiation.cc:299] Negotiation successful
0812 11:29:31.769699 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260812 11:29:31.770877 17564 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260812 11:29:31.771469 17564 leader_election.cc:302] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1c51c922d5747c3ac607ca88c5f5e25; no voters:
I20260812 11:29:31.772090 17564 leader_election.cc:288] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:31.772220 17583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:31.773005 17583 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [term 1 LEADER]: Becoming Leader. State: Replica: c1c51c922d5747c3ac607ca88c5f5e25, State: Running, Role: LEADER
I20260812 11:29:31.773600 17583 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260812 11:29:31.774712 17564 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:31.776386 17586 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1c51c922d5747c3ac607ca88c5f5e25. Latest consensus state: current_term: 1 leader_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260812 11:29:31.776839 17586 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:31.777957 17594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:31.778268 17585 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1c51c922d5747c3ac607ca88c5f5e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260812 11:29:31.778576 17585 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:31.784602 17594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:31.794094 17594 catalog_manager.cc:1347] Generated new cluster ID: bfb4026ffed2461f84b578e1729a6d88
I20260812 11:29:31.794234 17594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:31.802664 17594 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:31.804024 17594 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:31.817270 17594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1c51c922d5747c3ac607ca88c5f5e25: Generated new TSK 0
I20260812 11:29:31.818147 17594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:31.834486 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17454
W20260812 11:29:31.836365 17556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:35593
W20260812 11:29:31.837139 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17454: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:31.837302 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17454
W20260812 11:29:31.865594 17429 connection.cc:535] client connection to 127.0.0.1:41965 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41965): Connection reset by peer (error 104)
2026-08-12T11:29:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:31 dist-test-slave-h6n0 krb5kdc[17621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:31 dist-test-slave-h6n0 krb5kdc[17621](info): set up 2 sockets
Aug 12 11:29:31 dist-test-slave-h6n0 krb5kdc[17621](info): commencing operation
krb5kdc: starting...
W20260812 11:29:33.232710 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.347s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:33 dist-test-slave-h6n0 krb5kdc[17621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260812 11:29:33.583453 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:33.949678 17643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:33.949991 17643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:33.950208 17643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:33.957667 17643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:33.957796 17643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:33.957947 17643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:33.958068 17643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:33.958139 17643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:33.968520 17643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35497
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:33.969923 17643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:33.971393 17643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:33.983975 17649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:33.983939 17650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:33.983908 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:33.985109 17643 server_base.cc:1034] running on GCE node
I20260812 11:29:33.985888 17643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:33.987318 17643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:33.988535 17643 hybrid_clock.cc:648] HybridClock initialized: now 1786534173988483 us; error 43 us; skew 500 ppm
I20260812 11:29:33.988929 17643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:33 dist-test-slave-h6n0 krb5kdc[17621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:33.992394 17643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260812 11:29:33.994309 17643 webserver.cc:460] Webserver started at http://127.0.0.1:35441/ using document root <none> and password file <none>
I20260812 11:29:33.995069 17643 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:33.995180 17643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:33.995431 17643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:33.998499 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "06c60ecea1854350a3377f7a87c88b44"
format_stamp: "Formatted at 2026-08-12 11:29:33 on dist-test-slave-h6n0"
server_key: "b7395e35ae3f7b403b171c9436e9ec7a"
server_key_iv: "b6d76ed82c0c52df2df6535d800cd987"
server_key_version: "encryptionkey@0"
I20260812 11:29:33.999416 17643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "06c60ecea1854350a3377f7a87c88b44"
format_stamp: "Formatted at 2026-08-12 11:29:33 on dist-test-slave-h6n0"
server_key: "b7395e35ae3f7b403b171c9436e9ec7a"
server_key_iv: "b6d76ed82c0c52df2df6535d800cd987"
server_key_version: "encryptionkey@0"
I20260812 11:29:34.005138 17643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260812 11:29:34.008908 17665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:34.010283 17643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260812 11:29:34.010478 17643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "06c60ecea1854350a3377f7a87c88b44"
format_stamp: "Formatted at 2026-08-12 11:29:33 on dist-test-slave-h6n0"
server_key: "b7395e35ae3f7b403b171c9436e9ec7a"
server_key_iv: "b6d76ed82c0c52df2df6535d800cd987"
server_key_version: "encryptionkey@0"
I20260812 11:29:34.010757 17643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:34.016680 17643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:34.021178 17643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:34.021502 17643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:34.040954 17643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35497
I20260812 11:29:34.041009 17749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35497 every 8 connection(s)
I20260812 11:29:34.042418 17643 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:34.046254 17750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:34.050455 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17643
I20260812 11:29:34.050706 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:34.051185 12731 external_mini_cluster.cc:1428] Setting key 9d13741f8415516a113d36be1cc3c650
I20260812 11:29:34.057730 17750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44: Bootstrap starting.
I20260812 11:29:34.059726 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:34.053036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:33373 (local address 127.0.0.1:35497)
0812 11:29:34.053848 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:34.053862 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:34.053903 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:29:34.056297 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:34.056321 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:34.056645 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:34.056957 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:34.058098 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:34.058104 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:34.058116 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:34.058136 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:34.058197 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:34.058218 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:34.058407 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:34.058501 (+ 94us) server_negotiation.cc:299] Negotiation successful
0812 11:29:34.058741 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260812 11:29:34.061362 17750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:34.062520 17750 log.cc:824] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:34.063771 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:34.065243 17750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44: No bootstrap required, opened a new log
I20260812 11:29:34.070693 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:34.068245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.52:60905 (local address 127.0.0.1:35497)
0812 11:29:34.068399 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:34.068406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:34.068795 (+ 389us) server_negotiation.cc:407] Connection header received
0812 11:29:34.068944 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:34.068948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:34.068993 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:34.069069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:34.069991 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:34.069995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:34.069998 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:34.070016 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:34.070051 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:34.070069 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:34.070160 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:34.070291 (+ 131us) server_negotiation.cc:299] Negotiation successful
0812 11:29:34.070346 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260812 11:29:34.073796 17750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c60ecea1854350a3377f7a87c88b44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260812 11:29:34.074152 17750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:34.074267 17750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c60ecea1854350a3377f7a87c88b44, State: Initialized, Role: FOLLOWER
I20260812 11:29:34.074848 17750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c60ecea1854350a3377f7a87c88b44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260812 11:29:34.075096 17750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:34.075242 17750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:34.075420 17750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:34.077602 17750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c60ecea1854350a3377f7a87c88b44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260812 11:29:34.078073 17750 leader_election.cc:302] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c60ecea1854350a3377f7a87c88b44; no voters:
I20260812 11:29:34.078739 17750 leader_election.cc:288] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:34.078858 17766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:34.079682 17766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [term 1 LEADER]: Becoming Leader. State: Replica: 06c60ecea1854350a3377f7a87c88b44, State: Running, Role: LEADER
I20260812 11:29:34.080250 17766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c60ecea1854350a3377f7a87c88b44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } }
I20260812 11:29:34.080940 17750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:34.083614 17768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c60ecea1854350a3377f7a87c88b44. Latest consensus state: current_term: 1 leader_uuid: "06c60ecea1854350a3377f7a87c88b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c60ecea1854350a3377f7a87c88b44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } } }
I20260812 11:29:34.084133 17768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:34.083549 17767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c60ecea1854350a3377f7a87c88b44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c60ecea1854350a3377f7a87c88b44" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35497 } } }
I20260812 11:29:34.084792 17767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:34.085907 17779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:34.091348 17779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:34.099310 17779 catalog_manager.cc:1347] Generated new cluster ID: b41de71c124241d59cefc51a39ab2834
I20260812 11:29:34.099421 17779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:34.113214 17779 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:34.113631 17779 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:34.126245 17779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c60ecea1854350a3377f7a87c88b44: Generated new TSK 0
I20260812 11:29:34.127229 17779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:34.132351 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17643
W20260812 11:29:34.133903 17744 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.52:33373
W20260812 11:29:34.134661 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17643: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:34.134841 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17643
2026-08-12T11:29:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:34 dist-test-slave-h6n0 krb5kdc[17802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:34 dist-test-slave-h6n0 krb5kdc[17802](info): set up 2 sockets
Aug 12 11:29:34 dist-test-slave-h6n0 krb5kdc[17802](info): commencing operation
krb5kdc: starting...
W20260812 11:29:35.579661 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.394s user 0.002s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:35 dist-test-slave-h6n0 krb5kdc[17802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:35.949342 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:39753
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:44781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:39753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:36.297266 17825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:36.297540 17825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:36.297711 17825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:36.304847 17825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:36.304963 17825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:36.305011 17825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:36.305138 17825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:36.305210 17825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:36.315447 17825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:44781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:39753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:39753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:36.316860 17825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:36.318423 17825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:36.330585 17834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:36.330607 17836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:36.330886 17833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:36.332054 17825 server_base.cc:1034] running on GCE node
I20260812 11:29:36.332942 17825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:36.334576 17825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:36.335809 17825 hybrid_clock.cc:648] HybridClock initialized: now 1786534176335753 us; error 41 us; skew 500 ppm
I20260812 11:29:36.336153 17825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:36 dist-test-slave-h6n0 krb5kdc[17802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534176, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:36.339227 17825 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:36.340667 17825 webserver.cc:460] Webserver started at http://127.12.110.254:33159/ using document root <none> and password file <none>
I20260812 11:29:36.341586 17825 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:36.341732 17825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:36.342103 17825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:36.345340 17825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "cf52a3538b8a45049168a56bfd66cb11"
format_stamp: "Formatted at 2026-08-12 11:29:36 on dist-test-slave-h6n0"
server_key: "f695d37c8c02330c7c7b6cac4d818668"
server_key_iv: "54e150b27addc22759c62345335c17b9"
server_key_version: "encryptionkey@0"
I20260812 11:29:36.346146 17825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "cf52a3538b8a45049168a56bfd66cb11"
format_stamp: "Formatted at 2026-08-12 11:29:36 on dist-test-slave-h6n0"
server_key: "f695d37c8c02330c7c7b6cac4d818668"
server_key_iv: "54e150b27addc22759c62345335c17b9"
server_key_version: "encryptionkey@0"
I20260812 11:29:36.351326 17825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260812 11:29:36.355150 17848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:36.356489 17825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:29:36.356688 17825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "cf52a3538b8a45049168a56bfd66cb11"
format_stamp: "Formatted at 2026-08-12 11:29:36 on dist-test-slave-h6n0"
server_key: "f695d37c8c02330c7c7b6cac4d818668"
server_key_iv: "54e150b27addc22759c62345335c17b9"
server_key_version: "encryptionkey@0"
I20260812 11:29:36.356889 17825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:36.407676 17825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:36.412175 17825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:36.412478 17825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:36.430809 17825 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:39753
I20260812 11:29:36.430833 17931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:39753 every 8 connection(s)
I20260812 11:29:36.432150 17825 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:36.435546 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17825
I20260812 11:29:36.435758 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:36.436038 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:36.436321 12731 external_mini_cluster.cc:1428] Setting key dcbff956a62819265651468667abac42
I20260812 11:29:36.450893 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11: Bootstrap starting.
Aug 12 11:29:36 dist-test-slave-h6n0 krb5kdc[17802](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:36.454263 17933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:36.455427 17933 log.cc:824] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:36.458132 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11: No bootstrap required, opened a new log
I20260812 11:29:36.459903 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:36.438069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37309 (local address 127.12.110.254:39753)
0812 11:29:36.438879 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:36.438890 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:36.438932 (+ 42us) server_negotiation.cc:407] Connection header received
0812 11:29:36.440797 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:36.440817 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:36.441189 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:36.441480 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:36.443342 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:36.444611 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:36.446446 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:36.447192 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:36.447326 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:36.452382 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:36.452389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:36.452403 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:36.452442 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:36.455629 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:36.456466 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:36.456473 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:36.456478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:36.456635 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:36.457228 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:36.457235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:36.457244 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:36.457728 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:36.458066 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:36.458594 (+ 528us) server_negotiation.cc:299] Negotiation successful
0812 11:29:36.458922 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":395,"threads_started":1}
I20260812 11:29:36.464259 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:36.468679 17933 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf52a3538b8a45049168a56bfd66cb11" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39753 } }
I20260812 11:29:36.469139 17933 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:36.469262 17933 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf52a3538b8a45049168a56bfd66cb11, State: Initialized, Role: FOLLOWER
I20260812 11:29:36.470007 17933 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf52a3538b8a45049168a56bfd66cb11" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39753 } }
I20260812 11:29:36.470324 17933 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:36.470487 17933 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:36.470703 17933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:36.473176 17933 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf52a3538b8a45049168a56bfd66cb11" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39753 } }
I20260812 11:29:36.473722 17933 leader_election.cc:302] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf52a3538b8a45049168a56bfd66cb11; no voters:
I20260812 11:29:36.474359 17933 leader_election.cc:288] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:36.474519 17946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:36.475185 17946 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [term 1 LEADER]: Becoming Leader. State: Replica: cf52a3538b8a45049168a56bfd66cb11, State: Running, Role: LEADER
I20260812 11:29:36.475824 17946 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf52a3538b8a45049168a56bfd66cb11" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39753 } }
I20260812 11:29:36.478701 17948 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf52a3538b8a45049168a56bfd66cb11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf52a3538b8a45049168a56bfd66cb11" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39753 } } }
I20260812 11:29:36.478785 17947 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf52a3538b8a45049168a56bfd66cb11. Latest consensus state: current_term: 1 leader_uuid: "cf52a3538b8a45049168a56bfd66cb11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf52a3538b8a45049168a56bfd66cb11" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 39753 } } }
I20260812 11:29:36.479089 17933 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:36.479123 17948 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:36.479167 17947 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:36.482198 17953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:36.488662 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:36.469799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58977 (local address 127.12.110.254:39753)
0812 11:29:36.470040 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:36.470049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:36.470454 (+ 405us) server_negotiation.cc:407] Connection header received
0812 11:29:36.470740 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:36.470745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:36.470804 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:36.470912 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:36.472465 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:36.473611 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:36.475463 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:36.476161 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:36.476274 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:36.479539 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:36.479549 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:36.479555 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:36.479604 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:36.484056 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:36.485154 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:36.485159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:36.485163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:36.485302 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:36.485912 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:36.485917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:36.485920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:36.486397 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:36.486733 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:36.488080 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0812 11:29:36.488277 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:29:36.488678 17953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:36.501466 17953 catalog_manager.cc:1347] Generated new cluster ID: 2e435dc808cf4f03a4cb1ea89d96f75d
I20260812 11:29:36.501613 17953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:36.513675 17953 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:36.515857 17953 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:36.528716 17953 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf52a3538b8a45049168a56bfd66cb11: Generated new TSK 0
I20260812 11:29:36.529737 17953 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:36.547163 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17825
I20260812 11:29:36.548468 17922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:36.652796 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17825
W20260812 11:29:36.675041 17798 connection.cc:535] client connection to 127.12.110.254:39753 recv error: Network error: failed to read from TLS socket (remote: 127.12.110.254:39753): Connection reset by peer (error 104)
2026-08-12T11:29:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2509 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:36 dist-test-slave-h6n0 krb5kdc[17975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:36 dist-test-slave-h6n0 krb5kdc[17975](info): set up 2 sockets
Aug 12 11:29:36 dist-test-slave-h6n0 krb5kdc[17975](info): commencing operation
krb5kdc: starting...
W20260812 11:29:38.058949 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.365s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:38 dist-test-slave-h6n0 krb5kdc[17975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:38.445930 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:35531
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:33421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:35531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:38.802138 17990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:38.802424 17990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:38.802695 17990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:38.809938 17990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:38.810055 17990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:38.810178 17990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:38.810299 17990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:38.810387 17990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:38.820268 17990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:33421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:35531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:35531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:38.821666 17990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:38.823040 17990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:38.836499 17997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:38.836325 18001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:38.836444 17998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:38.837661 17990 server_base.cc:1034] running on GCE node
I20260812 11:29:38.838490 17990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:38.840009 17990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:38.841259 17990 hybrid_clock.cc:648] HybridClock initialized: now 1786534178841221 us; error 53 us; skew 500 ppm
I20260812 11:29:38.841621 17990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:38 dist-test-slave-h6n0 krb5kdc[17975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534178, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:38.845039 17990 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:38.846735 17990 webserver.cc:460] Webserver started at http://127.12.110.254:42447/ using document root <none> and password file <none>
I20260812 11:29:38.847435 17990 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:38.847579 17990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:38.847903 17990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:38.851004 17990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "807755f23c994417848e298b9509e6e4"
format_stamp: "Formatted at 2026-08-12 11:29:38 on dist-test-slave-h6n0"
server_key: "2bb4c6faadc8ba2d381490a42744163b"
server_key_iv: "39bbe3cc28891329bde8afbe4e820d5b"
server_key_version: "encryptionkey@0"
I20260812 11:29:38.851776 17990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "807755f23c994417848e298b9509e6e4"
format_stamp: "Formatted at 2026-08-12 11:29:38 on dist-test-slave-h6n0"
server_key: "2bb4c6faadc8ba2d381490a42744163b"
server_key_iv: "39bbe3cc28891329bde8afbe4e820d5b"
server_key_version: "encryptionkey@0"
I20260812 11:29:38.856918 17990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:29:38.860514 18016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:38.861896 17990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:29:38.862107 17990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "807755f23c994417848e298b9509e6e4"
format_stamp: "Formatted at 2026-08-12 11:29:38 on dist-test-slave-h6n0"
server_key: "2bb4c6faadc8ba2d381490a42744163b"
server_key_iv: "39bbe3cc28891329bde8afbe4e820d5b"
server_key_version: "encryptionkey@0"
I20260812 11:29:38.862324 17990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:38.878075 17990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:38.882666 17990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:38.883009 17990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:38.904186 17990 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:35531
I20260812 11:29:38.904238 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:35531 every 8 connection(s)
I20260812 11:29:38.905754 17990 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:38.906092 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 17990
I20260812 11:29:38.906312 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:38.907034 12731 external_mini_cluster.cc:1428] Setting key 019eecd087e29007123eba8e0d6e3c11
I20260812 11:29:38.910781 18105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:38.922922 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:38.909135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37457 (local address 127.12.110.254:35531)
0812 11:29:38.909956 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:38.909973 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:38.910025 (+ 52us) server_negotiation.cc:407] Connection header received
0812 11:29:38.912469 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:38.912508 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:38.912983 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:38.913404 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:38.913945 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:38.916019 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:38.917512 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:38.918515 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:38.918710 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:38.919963 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:38.919975 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:38.919992 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:38.920753 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:38.920846 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:38.920879 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:38.921066 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:38.921392 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 11:29:38.921586 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":351,"threads_started":1}
I20260812 11:29:38.925230 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4: Bootstrap starting.
I20260812 11:29:38.928580 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:38.929273 18105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:38.930413 18105 log.cc:824] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:38.933774 18105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4: No bootstrap required, opened a new log
I20260812 11:29:38.943065 18110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:38.934875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39137 (local address 127.12.110.254:35531)
0812 11:29:38.935098 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:38.935107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:38.935601 (+ 494us) server_negotiation.cc:407] Connection header received
0812 11:29:38.935918 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:38.935922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:38.935991 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:38.936081 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:38.936823 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:38.937645 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:38.939655 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:38.940237 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:38.940381 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:38.941914 (+ 1533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:38.941925 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:38.941930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:38.941964 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:38.942067 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:38.942091 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:38.942369 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:38.942596 (+ 227us) server_negotiation.cc:299] Negotiation successful
0812 11:29:38.942681 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:29:38.944605 18105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807755f23c994417848e298b9509e6e4" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35531 } }
I20260812 11:29:38.945115 18105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:38.945221 18105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807755f23c994417848e298b9509e6e4, State: Initialized, Role: FOLLOWER
I20260812 11:29:38.945806 18105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807755f23c994417848e298b9509e6e4" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35531 } }
I20260812 11:29:38.946031 18105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:38.946128 18105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:38.946295 18105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:38.949134 18105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807755f23c994417848e298b9509e6e4" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35531 } }
I20260812 11:29:38.949656 18105 leader_election.cc:302] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 807755f23c994417848e298b9509e6e4; no voters:
I20260812 11:29:38.950305 18105 leader_election.cc:288] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:38.950716 18123 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:38.951865 18123 raft_consensus.cc:686] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [term 1 LEADER]: Becoming Leader. State: Replica: 807755f23c994417848e298b9509e6e4, State: Running, Role: LEADER
I20260812 11:29:38.952477 18123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807755f23c994417848e298b9509e6e4" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35531 } }
I20260812 11:29:38.953336 18105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:38.955904 18125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 807755f23c994417848e298b9509e6e4. Latest consensus state: current_term: 1 leader_uuid: "807755f23c994417848e298b9509e6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807755f23c994417848e298b9509e6e4" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35531 } } }
I20260812 11:29:38.956281 18125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:38.957427 18124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "807755f23c994417848e298b9509e6e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "807755f23c994417848e298b9509e6e4" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35531 } } }
I20260812 11:29:38.957882 18124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:38.960031 18130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:38.965337 18130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:38.974565 18130 catalog_manager.cc:1347] Generated new cluster ID: 1bbdc9d18d94481d8576df4358b936f9
I20260812 11:29:38.974695 18130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:38.981894 18130 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:38.983469 18130 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:38.993470 18130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 807755f23c994417848e298b9509e6e4: Generated new TSK 0
I20260812 11:29:38.994478 18130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:39.005446 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17990
W20260812 11:29:39.007529 18095 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37457
W20260812 11:29:39.008673 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 17990: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:39.009019 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 17990
W20260812 11:29:39.039961 17973 connection.cc:535] client connection to 127.12.110.254:35531 recv error: Network error: failed to read from TLS socket (remote: 127.12.110.254:35531): Connection reset by peer (error 104)
2026-08-12T11:29:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:39 dist-test-slave-h6n0 krb5kdc[18150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:39 dist-test-slave-h6n0 krb5kdc[18150](info): set up 2 sockets
Aug 12 11:29:39 dist-test-slave-h6n0 krb5kdc[18150](info): commencing operation
krb5kdc: starting...
W20260812 11:29:40.443531 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.383s user 0.000s sys 0.060s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:40 dist-test-slave-h6n0 krb5kdc[18150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:40.788120 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43501
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:43501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:41.142275 18172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:41.142704 18172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:41.142899 18172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:41.150460 18172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:41.150676 18172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:41.150784 18172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:41.150898 18172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:41.150990 18172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:41.160943 18172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:43501
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:41.162364 18172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:41.163727 18172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:41.175632 18180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:41.175637 18181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:41.175688 18184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:41.176513 18172 server_base.cc:1034] running on GCE node
I20260812 11:29:41.177338 18172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:41.178810 18172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:41.180071 18172 hybrid_clock.cc:648] HybridClock initialized: now 1786534181180009 us; error 43 us; skew 500 ppm
I20260812 11:29:41.180481 18172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:41 dist-test-slave-h6n0 krb5kdc[18150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534181, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:41.183799 18172 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:41.185371 18172 webserver.cc:460] Webserver started at http://127.12.110.254:43541/ using document root <none> and password file <none>
I20260812 11:29:41.186050 18172 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:41.186206 18172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:41.186520 18172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:41.189574 18172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "04096af592d043bcbe750c51a9a60764"
format_stamp: "Formatted at 2026-08-12 11:29:41 on dist-test-slave-h6n0"
server_key: "d2a2a915d0076627ed2aec51dd03964e"
server_key_iv: "1c79688cfdb4f1760456e6f35eead519"
server_key_version: "encryptionkey@0"
I20260812 11:29:41.190356 18172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "04096af592d043bcbe750c51a9a60764"
format_stamp: "Formatted at 2026-08-12 11:29:41 on dist-test-slave-h6n0"
server_key: "d2a2a915d0076627ed2aec51dd03964e"
server_key_iv: "1c79688cfdb4f1760456e6f35eead519"
server_key_version: "encryptionkey@0"
I20260812 11:29:41.196274 18172 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:29:41.199837 18193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:41.201154 18172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:29:41.201359 18172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "04096af592d043bcbe750c51a9a60764"
format_stamp: "Formatted at 2026-08-12 11:29:41 on dist-test-slave-h6n0"
server_key: "d2a2a915d0076627ed2aec51dd03964e"
server_key_iv: "1c79688cfdb4f1760456e6f35eead519"
server_key_version: "encryptionkey@0"
I20260812 11:29:41.201589 18172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:41.207772 18172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:41.212003 18172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:41.212311 18172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:41.231125 18172 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:43501
I20260812 11:29:41.231172 18278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:43501 every 8 connection(s)
I20260812 11:29:41.232517 18172 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:41.235241 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18172
I20260812 11:29:41.235467 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:41.235940 12731 external_mini_cluster.cc:1428] Setting key f888833ffa2d4c0dc700c67bf729bc64
I20260812 11:29:41.236274 18279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:41.244897 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:41.237807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32869 (local address 127.12.110.254:43501)
0812 11:29:41.238672 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:41.238685 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:41.238738 (+ 53us) server_negotiation.cc:407] Connection header received
0812 11:29:41.241077 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:41.241104 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:41.241549 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:41.241969 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:41.242961 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:41.242969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:41.242986 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:41.243002 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:41.243081 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:41.243112 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:41.243346 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:41.243502 (+ 156us) server_negotiation.cc:299] Negotiation successful
0812 11:29:41.243685 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":325,"threads_started":1}
I20260812 11:29:41.248562 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:41.250823 18279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764: Bootstrap starting.
I20260812 11:29:41.254377 18279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:41.255522 18279 log.cc:824] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:41.256294 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:41.252987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57067 (local address 127.12.110.254:43501)
0812 11:29:41.253132 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:41.253136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:41.253660 (+ 524us) server_negotiation.cc:407] Connection header received
0812 11:29:41.253785 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:41.253791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:41.253848 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:41.253952 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:41.255484 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:41.255491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:41.255501 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:41.255523 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:41.255572 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:41.255593 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:41.255715 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:41.255860 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 11:29:41.255917 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260812 11:29:41.258874 18279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764: No bootstrap required, opened a new log
I20260812 11:29:41.266335 18279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04096af592d043bcbe750c51a9a60764" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43501 } }
I20260812 11:29:41.266732 18279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:41.266815 18279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04096af592d043bcbe750c51a9a60764, State: Initialized, Role: FOLLOWER
I20260812 11:29:41.267302 18279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04096af592d043bcbe750c51a9a60764" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43501 } }
I20260812 11:29:41.267498 18279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:41.267598 18279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:41.267719 18279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:41.270139 18279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04096af592d043bcbe750c51a9a60764" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43501 } }
I20260812 11:29:41.270689 18279 leader_election.cc:302] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04096af592d043bcbe750c51a9a60764; no voters:
I20260812 11:29:41.271297 18279 leader_election.cc:288] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:41.271466 18303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:41.272171 18303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [term 1 LEADER]: Becoming Leader. State: Replica: 04096af592d043bcbe750c51a9a60764, State: Running, Role: LEADER
I20260812 11:29:41.272697 18303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04096af592d043bcbe750c51a9a60764" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43501 } }
I20260812 11:29:41.273371 18279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:41.275807 18308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04096af592d043bcbe750c51a9a60764. Latest consensus state: current_term: 1 leader_uuid: "04096af592d043bcbe750c51a9a60764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04096af592d043bcbe750c51a9a60764" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43501 } } }
I20260812 11:29:41.276389 18308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:41.275892 18305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04096af592d043bcbe750c51a9a60764" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04096af592d043bcbe750c51a9a60764" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43501 } } }
I20260812 11:29:41.277158 18305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:41.284106 18317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:41.289674 18317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:41.301275 18317 catalog_manager.cc:1347] Generated new cluster ID: 035bb296334d48918379b8553b05b3a7
I20260812 11:29:41.301419 18317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:41.317675 18317 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:41.318145 18317 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:41.331189 18317 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04096af592d043bcbe750c51a9a60764: Generated new TSK 0
I20260812 11:29:41.332011 18317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:41.338688 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18172
W20260812 11:29:41.339905 18271 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32869
W20260812 11:29:41.340672 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18172: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:41.340907 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18172
2026-08-12T11:29:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:41 dist-test-slave-h6n0 krb5kdc[18336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:41 dist-test-slave-h6n0 krb5kdc[18336](info): set up 2 sockets
Aug 12 11:29:41 dist-test-slave-h6n0 krb5kdc[18336](info): commencing operation
krb5kdc: starting...
W20260812 11:29:42.770020 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.380s user 0.002s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:43 dist-test-slave-h6n0 krb5kdc[18336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:43.121786 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:35517
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:35517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:43.474754 18355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:43.475092 18355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:43.475278 18355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:43.483184 18355 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:43.483515 18355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:43.483644 18355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:43.483762 18355 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:43.483844 18355 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:43.494913 18355 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:35517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:35517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:43.496201 18355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:43.497608 18355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:43.510021 18361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:43.509913 18365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:43.509946 18367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:43.511179 18355 server_base.cc:1034] running on GCE node
I20260812 11:29:43.511960 18355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:43.513253 18355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:43.514712 18355 hybrid_clock.cc:648] HybridClock initialized: now 1786534183514519 us; error 34 us; skew 500 ppm
I20260812 11:29:43.515259 18355 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:43 dist-test-slave-h6n0 krb5kdc[18336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534183, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:43.523234 18355 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:43.524845 18355 webserver.cc:460] Webserver started at http://127.12.110.254:36389/ using document root <none> and password file <none>
I20260812 11:29:43.525544 18355 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:43.525698 18355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:43.526008 18355 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:43.529218 18355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "2a289565c9224915bf5dc48b1abfccd1"
format_stamp: "Formatted at 2026-08-12 11:29:43 on dist-test-slave-h6n0"
server_key: "4761ed0177045412ba42a52d46f015df"
server_key_iv: "275154bee9012333d6edce602a7a2d8c"
server_key_version: "encryptionkey@0"
I20260812 11:29:43.530020 18355 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "2a289565c9224915bf5dc48b1abfccd1"
format_stamp: "Formatted at 2026-08-12 11:29:43 on dist-test-slave-h6n0"
server_key: "4761ed0177045412ba42a52d46f015df"
server_key_iv: "275154bee9012333d6edce602a7a2d8c"
server_key_version: "encryptionkey@0"
I20260812 11:29:43.535380 18355 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:29:43.539096 18383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:43.540432 18355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260812 11:29:43.540639 18355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "2a289565c9224915bf5dc48b1abfccd1"
format_stamp: "Formatted at 2026-08-12 11:29:43 on dist-test-slave-h6n0"
server_key: "4761ed0177045412ba42a52d46f015df"
server_key_iv: "275154bee9012333d6edce602a7a2d8c"
server_key_version: "encryptionkey@0"
I20260812 11:29:43.540856 18355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:43.573137 18355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:43.578487 18355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:43.578835 18355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:43.600925 18355 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:35517
I20260812 11:29:43.600939 18461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:35517 every 8 connection(s)
I20260812 11:29:43.602380 18355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:43.605157 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18355
I20260812 11:29:43.605347 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:43.605928 12731 external_mini_cluster.cc:1428] Setting key 6d4bc72b5d2e7e3890688f076cda3ff5
I20260812 11:29:43.606783 18463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:43.620026 18463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1: Bootstrap starting.
Aug 12 11:29:43 dist-test-slave-h6n0 krb5kdc[18336](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:43.623526 18463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:43.624800 18463 log.cc:824] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:43.627841 18463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1: No bootstrap required, opened a new log
I20260812 11:29:43.630793 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:43.608329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44157 (local address 127.12.110.254:35517)
0812 11:29:43.609147 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:43.609159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:43.609200 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:29:43.611600 (+ 2400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:43.611621 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:43.612016 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:43.612417 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:43.614048 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:43.615865 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:43.617178 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:43.617869 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:43.618080 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:43.622476 (+ 4396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:43.622484 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:43.622495 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:43.622580 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:43.625898 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:43.626612 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:43.626640 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:43.626648 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:43.626951 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:43.627531 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:43.627537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:43.627540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:43.628191 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:43.628539 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:43.629013 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 11:29:43.629603 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":362,"threads_started":1}
I20260812 11:29:43.634759 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:43.636997 18463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a289565c9224915bf5dc48b1abfccd1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35517 } }
I20260812 11:29:43.637455 18463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:43.637576 18463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a289565c9224915bf5dc48b1abfccd1, State: Initialized, Role: FOLLOWER
I20260812 11:29:43.638260 18463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a289565c9224915bf5dc48b1abfccd1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35517 } }
I20260812 11:29:43.638671 18463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:43.638899 18463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:43.639109 18463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:43.642061 18463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a289565c9224915bf5dc48b1abfccd1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35517 } }
I20260812 11:29:43.642663 18463 leader_election.cc:302] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a289565c9224915bf5dc48b1abfccd1; no voters:
I20260812 11:29:43.643342 18463 leader_election.cc:288] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:43.643447 18483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:43.645915 18463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:43.651000 18483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [term 1 LEADER]: Becoming Leader. State: Replica: 2a289565c9224915bf5dc48b1abfccd1, State: Running, Role: LEADER
I20260812 11:29:43.651669 18483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a289565c9224915bf5dc48b1abfccd1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35517 } }
I20260812 11:29:43.655911 18484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a289565c9224915bf5dc48b1abfccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a289565c9224915bf5dc48b1abfccd1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35517 } } }
I20260812 11:29:43.656389 18484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:43.657002 18485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a289565c9224915bf5dc48b1abfccd1. Latest consensus state: current_term: 1 leader_uuid: "2a289565c9224915bf5dc48b1abfccd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a289565c9224915bf5dc48b1abfccd1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 35517 } } }
I20260812 11:29:43.657994 18485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:43.659116 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:43.640896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44043 (local address 127.12.110.254:35517)
0812 11:29:43.641108 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:43.641116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:43.641678 (+ 562us) server_negotiation.cc:407] Connection header received
0812 11:29:43.641960 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:43.641966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:43.642072 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:43.642320 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:43.644229 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:43.645187 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:43.646989 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:43.647438 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:43.647581 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:43.651316 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:43.651322 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:43.651326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:43.651359 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:43.655415 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:43.656395 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:43.656401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:43.656404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:43.656583 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:43.657118 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:43.657126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:43.657129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:43.657634 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:43.657936 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:43.658645 (+ 709us) server_negotiation.cc:299] Negotiation successful
0812 11:29:43.658841 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:29:43.659484 18497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:43.665666 18497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:43.675329 18497 catalog_manager.cc:1347] Generated new cluster ID: 3222d60e41eb4400844a564f0bacfd5e
I20260812 11:29:43.675498 18497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:43.686549 18497 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:43.688455 18497 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:43.713675 18497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a289565c9224915bf5dc48b1abfccd1: Generated new TSK 0
I20260812 11:29:43.714761 18497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:43.724462 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18355
I20260812 11:29:43.725525 18455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:43.818634 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18355
2026-08-12T11:29:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2472 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:43 dist-test-slave-h6n0 krb5kdc[18513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:43 dist-test-slave-h6n0 krb5kdc[18513](info): set up 2 sockets
Aug 12 11:29:43 dist-test-slave-h6n0 krb5kdc[18513](info): commencing operation
krb5kdc: starting...
W20260812 11:29:45.206920 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.000s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:45 dist-test-slave-h6n0 krb5kdc[18513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:45.539091 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:34651
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:45.893813 18538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:45.894096 18538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:45.894273 18538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:45.901400 18538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:45.901499 18538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:45.901608 18538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:45.901696 18538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:45.901762 18538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:45.911149 18538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:34651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:45.912336 18538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:45.913618 18538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:45.925577 18544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:45.925534 18547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:45.925607 18549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:45.926563 18538 server_base.cc:1034] running on GCE node
I20260812 11:29:45.927392 18538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:45.928730 18538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:45.929932 18538 hybrid_clock.cc:648] HybridClock initialized: now 1786534185929903 us; error 30 us; skew 500 ppm
I20260812 11:29:45.930267 18538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:45 dist-test-slave-h6n0 krb5kdc[18513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534185, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:45.933511 18538 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:45.935093 18538 webserver.cc:460] Webserver started at http://127.12.110.254:38433/ using document root <none> and password file <none>
I20260812 11:29:45.935778 18538 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:45.935885 18538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:45.936131 18538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:45.939102 18538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "0eea01524e694b6dbe18162d2ee44630"
format_stamp: "Formatted at 2026-08-12 11:29:45 on dist-test-slave-h6n0"
server_key: "8805e35b7b41457057387dc90358c8b5"
server_key_iv: "a1181e38f81f34d26761d77789cee7b9"
server_key_version: "encryptionkey@0"
I20260812 11:29:45.939826 18538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "0eea01524e694b6dbe18162d2ee44630"
format_stamp: "Formatted at 2026-08-12 11:29:45 on dist-test-slave-h6n0"
server_key: "8805e35b7b41457057387dc90358c8b5"
server_key_iv: "a1181e38f81f34d26761d77789cee7b9"
server_key_version: "encryptionkey@0"
I20260812 11:29:45.945519 18538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260812 11:29:45.949162 18560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:45.950443 18538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260812 11:29:45.950692 18538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "0eea01524e694b6dbe18162d2ee44630"
format_stamp: "Formatted at 2026-08-12 11:29:45 on dist-test-slave-h6n0"
server_key: "8805e35b7b41457057387dc90358c8b5"
server_key_iv: "a1181e38f81f34d26761d77789cee7b9"
server_key_version: "encryptionkey@0"
I20260812 11:29:45.950896 18538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:45.967083 18538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:45.971835 18538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:45.972146 18538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:45.994102 18538 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:34651
I20260812 11:29:45.994136 18644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:34651 every 8 connection(s)
I20260812 11:29:45.995697 18538 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:45.999271 18648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:46.004285 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18538
I20260812 11:29:46.004491 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:46.005066 12731 external_mini_cluster.cc:1428] Setting key a22fc971516b6f5a7d1257e32972e29f
I20260812 11:29:46.010812 18648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630: Bootstrap starting.
I20260812 11:29:46.013787 18648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:46.015285 18648 log.cc:824] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:46.019042 18648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630: No bootstrap required, opened a new log
I20260812 11:29:46.019645 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:46.006940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36205 (local address 127.12.110.254:34651)
0812 11:29:46.007702 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:46.007714 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:46.007752 (+ 38us) server_negotiation.cc:407] Connection header received
0812 11:29:46.009528 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:46.009550 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:46.009886 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:46.010235 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:46.010776 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:46.012100 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:46.013406 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:46.014136 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:46.014587 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:46.016468 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:46.016477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:46.016494 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:46.017199 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:46.017313 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:46.017358 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:46.017501 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:46.017975 (+ 474us) server_negotiation.cc:299] Negotiation successful
0812 11:29:46.018253 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
I20260812 11:29:46.024124 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:46.030388 18648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea01524e694b6dbe18162d2ee44630" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 34651 } }
I20260812 11:29:46.030901 18648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:46.031038 18648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eea01524e694b6dbe18162d2ee44630, State: Initialized, Role: FOLLOWER
I20260812 11:29:46.031656 18648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea01524e694b6dbe18162d2ee44630" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 34651 } }
I20260812 11:29:46.031957 18648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:46.032117 18648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:46.032305 18648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:46.035091 18648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea01524e694b6dbe18162d2ee44630" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 34651 } }
I20260812 11:29:46.035642 18648 leader_election.cc:302] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eea01524e694b6dbe18162d2ee44630; no voters:
I20260812 11:29:46.036307 18648 leader_election.cc:288] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:46.036449 18663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:46.038187 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:46.029814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34929 (local address 127.12.110.254:34651)
0812 11:29:46.030007 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:46.030013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:46.030585 (+ 572us) server_negotiation.cc:407] Connection header received
0812 11:29:46.030788 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:46.030794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:46.030880 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:46.031003 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:46.031663 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:46.032773 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:46.034099 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:46.034816 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:46.034956 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:46.036582 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:46.036588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:46.036593 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:46.036620 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:46.036679 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:46.036725 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:46.037450 (+ 725us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:46.037519 (+ 69us) server_negotiation.cc:299] Negotiation successful
0812 11:29:46.037648 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:29:46.038491 18648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:46.042959 18663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [term 1 LEADER]: Becoming Leader. State: Replica: 0eea01524e694b6dbe18162d2ee44630, State: Running, Role: LEADER
I20260812 11:29:46.043641 18663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea01524e694b6dbe18162d2ee44630" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 34651 } }
I20260812 11:29:46.046520 18666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eea01524e694b6dbe18162d2ee44630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea01524e694b6dbe18162d2ee44630" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 34651 } } }
I20260812 11:29:46.047591 18666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:46.046904 18670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eea01524e694b6dbe18162d2ee44630. Latest consensus state: current_term: 1 leader_uuid: "0eea01524e694b6dbe18162d2ee44630" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eea01524e694b6dbe18162d2ee44630" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 34651 } } }
I20260812 11:29:46.047767 18670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:46.051537 18681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:46.058686 18681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:46.067595 18681 catalog_manager.cc:1347] Generated new cluster ID: 08300fc7387a4b9daf876fb9b22ec8b7
I20260812 11:29:46.067708 18681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:46.076155 18681 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:46.077476 18681 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:46.087379 18681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eea01524e694b6dbe18162d2ee44630: Generated new TSK 0
I20260812 11:29:46.088109 18681 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:46.097669 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18538
W20260812 11:29:46.100636 18637 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36205
W20260812 11:29:46.101339 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18538: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:46.101478 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18538
2026-08-12T11:29:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2287 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:46 dist-test-slave-h6n0 krb5kdc[18701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:46 dist-test-slave-h6n0 krb5kdc[18701](info): set up 2 sockets
Aug 12 11:29:46 dist-test-slave-h6n0 krb5kdc[18701](info): commencing operation
krb5kdc: starting...
W20260812 11:29:47.504330 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.354s user 0.004s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:47 dist-test-slave-h6n0 krb5kdc[18701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:47.842212 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43689
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:48.194213 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:48.194612 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:48.194773 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:48.203127 18721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:48.203301 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:48.203416 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:48.203534 18721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:48.203611 18721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:48.213647 18721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:43689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:48.215056 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:48.216372 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:48.230145 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:48.230252 18729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:48.230226 18735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:48.231496 18721 server_base.cc:1034] running on GCE node
I20260812 11:29:48.232316 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:48.233769 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:48.235033 18721 hybrid_clock.cc:648] HybridClock initialized: now 1786534188234990 us; error 35 us; skew 500 ppm
I20260812 11:29:48.235441 18721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:48 dist-test-slave-h6n0 krb5kdc[18701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534188, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:48.239508 18721 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:48.241142 18721 webserver.cc:460] Webserver started at http://127.12.110.254:36749/ using document root <none> and password file <none>
I20260812 11:29:48.241840 18721 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:48.242003 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:48.242321 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:48.245383 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "3a7b184ccd6e47b7a39aba965909d3d1"
format_stamp: "Formatted at 2026-08-12 11:29:48 on dist-test-slave-h6n0"
server_key: "34444f3bbcc4d764343dc2c206847f64"
server_key_iv: "dbbcc9eeb3476eb3589df347830a4d4b"
server_key_version: "encryptionkey@0"
I20260812 11:29:48.246168 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "3a7b184ccd6e47b7a39aba965909d3d1"
format_stamp: "Formatted at 2026-08-12 11:29:48 on dist-test-slave-h6n0"
server_key: "34444f3bbcc4d764343dc2c206847f64"
server_key_iv: "dbbcc9eeb3476eb3589df347830a4d4b"
server_key_version: "encryptionkey@0"
I20260812 11:29:48.251659 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260812 11:29:48.255509 18750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:48.256894 18721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260812 11:29:48.257103 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "3a7b184ccd6e47b7a39aba965909d3d1"
format_stamp: "Formatted at 2026-08-12 11:29:48 on dist-test-slave-h6n0"
server_key: "34444f3bbcc4d764343dc2c206847f64"
server_key_iv: "dbbcc9eeb3476eb3589df347830a4d4b"
server_key_version: "encryptionkey@0"
I20260812 11:29:48.257316 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:48.263451 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:48.267599 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:48.267892 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:48.289613 18721 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:43689
I20260812 11:29:48.289700 18835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:43689 every 8 connection(s)
I20260812 11:29:48.291072 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:48.294763 18838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:48.301558 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18721
I20260812 11:29:48.301767 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:48.302529 12731 external_mini_cluster.cc:1428] Setting key 1e6e651196eefd4e1e17e8e82cae554e
I20260812 11:29:48.307637 18838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1: Bootstrap starting.
I20260812 11:29:48.311232 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:48.304631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51835 (local address 127.12.110.254:43689)
0812 11:29:48.305650 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:48.305663 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:48.305707 (+ 44us) server_negotiation.cc:407] Connection header received
0812 11:29:48.307516 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:48.307539 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:48.307936 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:48.308280 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:48.309483 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:48.309490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:48.309502 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:48.309520 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:48.309582 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:48.309611 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:48.309780 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:48.310006 (+ 226us) server_negotiation.cc:299] Negotiation successful
0812 11:29:48.310238 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":786,"thread_start_us":496,"threads_started":1}
I20260812 11:29:48.311792 18838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:48.312961 18838 log.cc:824] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:48.315250 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:48.316715 18838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1: No bootstrap required, opened a new log
I20260812 11:29:48.323366 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:48.320283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57687 (local address 127.12.110.254:43689)
0812 11:29:48.320485 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:48.320492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:48.321078 (+ 586us) server_negotiation.cc:407] Connection header received
0812 11:29:48.321279 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:48.321284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:48.321335 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:48.321439 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:48.322525 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:48.322603 (+ 78us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:48.322607 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0812 11:29:48.322624 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:48.322663 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0812 11:29:48.322683 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:48.322805 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:48.322962 (+ 157us) server_negotiation.cc:299] Negotiation successful
0812 11:29:48.323020 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260812 11:29:48.326982 18838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43689 } }
I20260812 11:29:48.327410 18838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260812 11:29:48.327529 18838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7b184ccd6e47b7a39aba965909d3d1, State: Initialized, Role: FOLLOWER
I20260812 11:29:48.328136 18838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43689 } }
I20260812 11:29:48.328408 18838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260812 11:29:48.328572 18838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260812 11:29:48.328763 18838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:48.331161 18838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43689 } }
I20260812 11:29:48.331743 18838 leader_election.cc:302] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a7b184ccd6e47b7a39aba965909d3d1; no voters:
I20260812 11:29:48.332340 18838 leader_election.cc:288] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260812 11:29:48.332512 18853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:48.333225 18853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7b184ccd6e47b7a39aba965909d3d1, State: Running, Role: LEADER
I20260812 11:29:48.333787 18853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43689 } }
I20260812 11:29:48.334417 18838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:48.336923 18860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a7b184ccd6e47b7a39aba965909d3d1. Latest consensus state: current_term: 1 leader_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43689 } } }
I20260812 11:29:48.337265 18859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7b184ccd6e47b7a39aba965909d3d1" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 43689 } } }
I20260812 11:29:48.337620 18859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:48.337620 18860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:48.338776 18866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:48.346860 18866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:48.356704 18866 catalog_manager.cc:1347] Generated new cluster ID: 2fb49aa1f4244c1082eed7def4bce7fa
I20260812 11:29:48.356844 18866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:48.373751 18866 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:48.374251 18866 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:48.390986 18866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a7b184ccd6e47b7a39aba965909d3d1: Generated new TSK 0
I20260812 11:29:48.391817 18866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:48.407074 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18721
W20260812 11:29:48.408247 18827 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51835
W20260812 11:29:48.408974 12731 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260812 11:29:48.409124 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18721
2026-08-12T11:29:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2306 ms)
[----------] 18 tests from AuthTokenIssuingTest (42716 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:48 dist-test-slave-h6n0 krb5kdc[18883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:48 dist-test-slave-h6n0 krb5kdc[18883](info): set up 2 sockets
Aug 12 11:29:48 dist-test-slave-h6n0 krb5kdc[18883](info): commencing operation
krb5kdc: starting...
W20260812 11:29:49.814646 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.358s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:50 dist-test-slave-h6n0 krb5kdc[18883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:50.155802 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:46289
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:46289,127.12.110.253:46775,127.12.110.252:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:50.504271 18904 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:29:50.504603 18904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:50.504678 18904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:50.504765 18904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:50.512213 18904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:50.512305 18904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:50.512363 18904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:50.512413 18904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:50.512475 18904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:50.522305 18904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:46289,127.12.110.253:46775,127.12.110.252:32985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:46289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:50.523639 18904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:50.524878 18904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:50.537433 18915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:50.537371 18920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:50.537406 18918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:50.538741 18904 server_base.cc:1034] running on GCE node
I20260812 11:29:50.539602 18904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:50.540992 18904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:50.542270 18904 hybrid_clock.cc:648] HybridClock initialized: now 1786534190542220 us; error 46 us; skew 500 ppm
I20260812 11:29:50.542711 18904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:50 dist-test-slave-h6n0 krb5kdc[18883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:50.550263 18904 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:50.551788 18904 webserver.cc:460] Webserver started at http://127.12.110.254:46541/ using document root <none> and password file <none>
I20260812 11:29:50.552510 18904 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:50.552635 18904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:50.552996 18904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:50.556136 18904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "40d73cee3f734bb99862d846e96c5210"
format_stamp: "Formatted at 2026-08-12 11:29:50 on dist-test-slave-h6n0"
server_key: "b3b289b6b08f23969ad654eead909029"
server_key_iv: "a05c5420b7e9f494bacb2942ebba394f"
server_key_version: "encryptionkey@0"
I20260812 11:29:50.557018 18904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "40d73cee3f734bb99862d846e96c5210"
format_stamp: "Formatted at 2026-08-12 11:29:50 on dist-test-slave-h6n0"
server_key: "b3b289b6b08f23969ad654eead909029"
server_key_iv: "a05c5420b7e9f494bacb2942ebba394f"
server_key_version: "encryptionkey@0"
I20260812 11:29:50.562160 18904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:29:50.565640 18933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:50.567103 18904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260812 11:29:50.567303 18904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "40d73cee3f734bb99862d846e96c5210"
format_stamp: "Formatted at 2026-08-12 11:29:50 on dist-test-slave-h6n0"
server_key: "b3b289b6b08f23969ad654eead909029"
server_key_iv: "a05c5420b7e9f494bacb2942ebba394f"
server_key_version: "encryptionkey@0"
I20260812 11:29:50.567670 18904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:50.583130 18904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:50.587641 18904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:50.587925 18904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:50.608023 18904 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:46289
I20260812 11:29:50.608071 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:46289 every 8 connection(s)
I20260812 11:29:50.609510 18904 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:50.613193 19008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:50.617779 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 18904
I20260812 11:29:50.618014 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:50.618907 12731 external_mini_cluster.cc:1428] Setting key 9998a39c9aa509bcb0fc7ec487baba03
I20260812 11:29:50.620915 19008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:50 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:50.645218 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.625832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45758 (local address 127.12.110.254:46289)
0812 11:29:50.626560 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:50.626570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:50.626624 (+ 54us) server_negotiation.cc:407] Connection header received
0812 11:29:50.627897 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:50.627907 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:50.628195 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:50.628414 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:50.629558 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:50.631441 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:50.633182 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:50.634092 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:50.634276 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:50.637919 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:50.637926 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:50.637937 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:50.637964 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:50.640665 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:50.641313 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:50.641321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:50.641327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:50.641466 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:50.641856 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:50.641861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:50.641869 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:50.642308 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:50.642502 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:50.643038 (+ 536us) server_negotiation.cc:299] Negotiation successful
0812 11:29:50.643422 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":483,"thread_start_us":331,"threads_started":1}
I20260812 11:29:50.645241 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.624578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:46289 (local address 127.0.0.1:45758)
0812 11:29:50.625397 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.625443 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:50.627546 (+ 2103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:50.628446 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:50.628450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:50.629009 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:50.629333 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:50.629349 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:50.631627 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:50.631631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:50.633015 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:50.633032 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:50.634316 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:50.634324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:50.634658 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:50.635494 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:50.635518 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:50.637684 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:50.640795 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:50.640808 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:50.640818 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:50.641185 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:50.641576 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:50.641582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:50.641586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:50.641743 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:50.642443 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:50.642449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:50.642661 (+ 212us) client_negotiation.cc:769] Sending connection context
0812 11:29:50.642910 (+ 249us) client_negotiation.cc:240] Negotiation successful
0812 11:29:50.643537 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":247,"thread_start_us":369,"threads_started":1}
I20260812 11:29:50.648939 19008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:50.650167 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.649647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49074)
0812 11:29:50.649961 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.650059 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
W20260812 11:29:50.650453 18939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
I20260812 11:29:50.652070 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.651722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49084)
0812 11:29:50.651868 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.651940 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:29:50.652298 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:46775: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.12.110.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab.
Entry for principal kudu/127.12.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab.
I20260812 11:29:50.697604 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:46775
--webserver_interface=127.12.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab
--principal=kudu/127.12.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:46289,127.12.110.253:46775,127.12.110.252:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:29:50.701619 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } attempt: 1
I20260812 11:29:50.702917 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.702408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49094)
0812 11:29:50.702660 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.702784 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 11:29:50.704036 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.703677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49106)
0812 11:29:50.703821 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.703893 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260812 11:29:50.704306 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:46775: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
I20260812 11:29:50.772753 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } attempt: 2
I20260812 11:29:50.774087 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.773544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49122)
0812 11:29:50.773836 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.773932 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260812 11:29:50.775323 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.774915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49138)
0812 11:29:50.775112 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.775192 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260812 11:29:50.775583 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:46775: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
I20260812 11:29:50.866904 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } attempt: 3
I20260812 11:29:50.868119 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.867665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49152)
0812 11:29:50.867868 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.867967 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 11:29:50.869295 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:50.868916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49154)
0812 11:29:50.869096 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:50.869165 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260812 11:29:50.869506 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:46775: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
I20260812 11:29:51.012780 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } attempt: 4
I20260812 11:29:51.014458 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.013558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49164)
0812 11:29:51.014028 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.014310 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260812 11:29:51.015686 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.015290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49174)
0812 11:29:51.015455 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.015562 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260812 11:29:51.015926 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:46775: Network error: Client connection negotiation failed: client connection to 127.12.110.253:46775: connect: Connection refused (error 111)
W20260812 11:29:51.054296 19019 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:29:51.054762 19019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:51.054863 19019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:51.054998 19019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:51.062819 19019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:51.062917 19019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:51.063024 19019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:51.063113 19019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:51.063205 19019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:51.073889 19019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:46289,127.12.110.253:46775,127.12.110.252:32985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab
--principal=kudu/127.12.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:51.075304 19019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:51.076683 19019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:51.089469 19025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:51.089736 19026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:51.089771 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:51.090994 19019 server_base.cc:1034] running on GCE node
I20260812 11:29:51.092054 19019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:51.093551 19019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:51.094820 19019 hybrid_clock.cc:648] HybridClock initialized: now 1786534191094699 us; error 74 us; skew 500 ppm
I20260812 11:29:51.095206 19019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:51.099383 19019 init.cc:377] Logged in from keytab as kudu/127.12.110.253@KRBTEST.COM (short username kudu)
I20260812 11:29:51.101357 19019 webserver.cc:460] Webserver started at http://127.12.110.253:36053/ using document root <none> and password file <none>
I20260812 11:29:51.102146 19019 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:51.102294 19019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:51.102718 19019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:51.105916 19019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data/instance:
uuid: "a9a2f320947d42fa843031bbaa5c9996"
format_stamp: "Formatted at 2026-08-12 11:29:51 on dist-test-slave-h6n0"
server_key: "a5b787ae3263cfd6727d2a86a34b6e4e"
server_key_iv: "fba37b97122e2a7c3906a02c93a704bf"
server_key_version: "encryptionkey@0"
I20260812 11:29:51.106873 19019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/wal/instance:
uuid: "a9a2f320947d42fa843031bbaa5c9996"
format_stamp: "Formatted at 2026-08-12 11:29:51 on dist-test-slave-h6n0"
server_key: "a5b787ae3263cfd6727d2a86a34b6e4e"
server_key_iv: "fba37b97122e2a7c3906a02c93a704bf"
server_key_version: "encryptionkey@0"
I20260812 11:29:51.112461 19019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260812 11:29:51.116137 19043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:51.117548 19019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260812 11:29:51.117762 19019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/wal
uuid: "a9a2f320947d42fa843031bbaa5c9996"
format_stamp: "Formatted at 2026-08-12 11:29:51 on dist-test-slave-h6n0"
server_key: "a5b787ae3263cfd6727d2a86a34b6e4e"
server_key_iv: "fba37b97122e2a7c3906a02c93a704bf"
server_key_version: "encryptionkey@0"
I20260812 11:29:51.118010 19019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:51.136660 19019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:51.141422 19019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:51.141722 19019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:51.163385 19019 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.253:46775
I20260812 11:29:51.163421 19113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.253:46775 every 8 connection(s)
I20260812 11:29:51.164798 19019 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
I20260812 11:29:51.168972 19114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:51.175047 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19019
I20260812 11:29:51.175261 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-1/wal/instance
I20260812 11:29:51.175727 12731 external_mini_cluster.cc:1428] Setting key 8f9dad841849e5fc585700ac89614464
I20260812 11:29:51.176350 19114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:51.201236 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.179020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.12.110.254:46289)
0812 11:29:51.179627 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.179635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.180635 (+ 1000us) server_negotiation.cc:407] Connection header received
0812 11:29:51.183465 (+ 2830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.183470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.183541 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.183649 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.185709 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.186600 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.189057 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.189480 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.189599 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.193820 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.193829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.193836 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.193886 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.197808 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.198567 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.198575 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.198581 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.198737 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.199184 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.199192 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.199198 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.199668 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.199846 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.200467 (+ 621us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.200726 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":319,"threads_started":1}
I20260812 11:29:51.201735 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.179569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:46289 (local address 127.0.0.1:45760)
0812 11:29:51.180443 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.180494 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.183172 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.183907 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.183919 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.184694 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.185476 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.185502 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.186790 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.186799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.188859 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.188892 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.189714 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.189723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.190066 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.190893 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.190928 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.193583 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.197951 (+ 4368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.197958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.197968 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.198405 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.198880 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.198883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.198886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.199064 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.199810 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.199815 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.200111 (+ 296us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.200332 (+ 221us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.200684 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":333,"threads_started":1}
I20260812 11:29:51.203634 19114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.252@KRBTEST.COM" created.
I20260812 11:29:51.219296 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.204530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49180)
0812 11:29:51.204685 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.204711 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.204823 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.205549 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.205554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.205982 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.206100 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.206108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.207528 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.207534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.208362 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.208373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.209279 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.209284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.209478 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.210039 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.210055 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.212409 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.216493 (+ 4084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.216499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.216506 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.216946 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.217385 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.217392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.217398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.217564 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.218189 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.218193 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.218343 (+ 150us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.218517 (+ 174us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.218856 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260812 11:29:51.219429 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.204613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49180 (local address 127.12.110.253:46775)
0812 11:29:51.205186 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.205192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.205215 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:29:51.205256 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.205261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.205401 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.205564 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.206277 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.207385 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.208505 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.209138 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.209273 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.212628 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.212639 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.212651 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.212677 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.216303 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.217101 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.217110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.217114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.217255 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.217680 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.217687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.217691 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.218051 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.218291 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.218703 (+ 412us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.218982 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":431,"thread_start_us":280,"threads_started":1}
I20260812 11:29:51.221962 19114 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:51.223307 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.222781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:42996)
0812 11:29:51.223039 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.223155 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260812 11:29:51.223592 19044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.225579 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.225196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43004)
0812 11:29:51.225396 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.225473 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 11:29:51.225862 19114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab.
Entry for principal kudu/127.12.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab.
I20260812 11:29:51.257856 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:32985
--webserver_interface=127.12.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab
--principal=kudu/127.12.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:36393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:46289,127.12.110.253:46775,127.12.110.252:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:29:51.275210 19114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 1
I20260812 11:29:51.276539 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.275902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43020)
0812 11:29:51.276170 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.276327 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260812 11:29:51.277654 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.277175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43022)
0812 11:29:51.277400 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.277504 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 11:29:51.277886 19114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.315235 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } attempt: 5
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:29:51.331084 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.316080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49190)
0812 11:29:51.316279 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.316326 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.316473 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.316859 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.316865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.317326 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.317518 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.317527 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.318572 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.318579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.319431 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.319442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.320235 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.320241 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.320480 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.321273 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.321293 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.324062 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.328493 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.328499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.328504 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.328926 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.329325 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.329331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.329334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.329473 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.330035 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.330040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.330203 (+ 163us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.330352 (+ 149us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.330725 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260812 11:29:51.331084 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.316250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.12.110.253:46775)
0812 11:29:51.316458 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.316463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.316486 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:29:51.316624 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.316630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.316705 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.316819 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.317693 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.318403 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.319598 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.320066 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.320206 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.324264 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.324269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.324272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.324303 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.328312 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.329067 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.329073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.329076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.329210 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.329595 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.329602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.329605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.329913 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.330078 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.330463 (+ 385us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.330730 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:29:51.332118 19008 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:51.333214 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.332834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43030)
0812 11:29:51.333007 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.333097 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260812 11:29:51.334151 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.333812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43042)
0812 11:29:51.333965 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.334044 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 11:29:51.334373 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.346129 19114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 2
I20260812 11:29:51.347275 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.346736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43058)
0812 11:29:51.346932 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.347046 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260812 11:29:51.348774 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.348342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43060)
0812 11:29:51.348539 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.348636 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260812 11:29:51.349299 19114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.385598 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 1
I20260812 11:29:51.386956 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.386311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43076)
0812 11:29:51.386589 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.386715 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260812 11:29:51.388152 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.387762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43086)
0812 11:29:51.387926 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.388010 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 11:29:51.388417 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.440582 19114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 3
I20260812 11:29:51.442103 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.441436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43102)
0812 11:29:51.441705 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.441847 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 11:29:51.443406 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.442878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43112)
0812 11:29:51.443140 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.443247 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260812 11:29:51.443707 19114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.456655 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 2
I20260812 11:29:51.457877 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.457312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43118)
0812 11:29:51.457565 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.457668 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260812 11:29:51.459021 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.458638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43124)
0812 11:29:51.458786 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.458880 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260812 11:29:51.459257 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.565527 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 3
I20260812 11:29:51.567130 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.566396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43134)
0812 11:29:51.566724 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.566875 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260812 11:29:51.568567 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.568121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43142)
0812 11:29:51.568336 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.568437 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260812 11:29:51.568926 19008 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
I20260812 11:29:51.586946 19114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 4
I20260812 11:29:51.588198 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.587577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43144)
0812 11:29:51.587848 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.587963 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260812 11:29:51.589387 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.588984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43148)
0812 11:29:51.589150 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.589247 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 11:29:51.589749 19114 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:32985: Network error: Client connection negotiation failed: client connection to 127.12.110.252:32985: connect: Connection refused (error 111)
W20260812 11:29:51.620985 19124 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:29:51.621340 19124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:51.621461 19124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:51.621560 19124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:51.629818 19124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:51.629912 19124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:51.629976 19124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:51.630030 19124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:51.630089 19124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:51.647873 19124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:36393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:46289,127.12.110.253:46775,127.12.110.252:32985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab
--principal=kudu/127.12.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:51.649247 19124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:51.650475 19124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:51.662204 19138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:51.662288 19134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:51.662258 19131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:51.663210 19124 server_base.cc:1034] running on GCE node
I20260812 11:29:51.664110 19124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:51.665501 19124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:51.666778 19124 hybrid_clock.cc:648] HybridClock initialized: now 1786534191666720 us; error 34 us; skew 500 ppm
I20260812 11:29:51.667165 19124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:51.670492 19124 init.cc:377] Logged in from keytab as kudu/127.12.110.252@KRBTEST.COM (short username kudu)
I20260812 11:29:51.671998 19124 webserver.cc:460] Webserver started at http://127.12.110.252:40115/ using document root <none> and password file <none>
I20260812 11:29:51.672744 19124 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:51.672859 19124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:51.673236 19124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:51.676429 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data/instance:
uuid: "c32930a11ee241b885a35e96eb819ffe"
format_stamp: "Formatted at 2026-08-12 11:29:51 on dist-test-slave-h6n0"
server_key: "d4769e25a75d6499f0d751ded272d151"
server_key_iv: "7ca76e6905305b4d669803f8d71e0424"
server_key_version: "encryptionkey@0"
I20260812 11:29:51.677222 19124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/wal/instance:
uuid: "c32930a11ee241b885a35e96eb819ffe"
format_stamp: "Formatted at 2026-08-12 11:29:51 on dist-test-slave-h6n0"
server_key: "d4769e25a75d6499f0d751ded272d151"
server_key_iv: "7ca76e6905305b4d669803f8d71e0424"
server_key_version: "encryptionkey@0"
I20260812 11:29:51.682399 19124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260812 11:29:51.685987 19146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:51.687321 19124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260812 11:29:51.687525 19124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/wal
uuid: "c32930a11ee241b885a35e96eb819ffe"
format_stamp: "Formatted at 2026-08-12 11:29:51 on dist-test-slave-h6n0"
server_key: "d4769e25a75d6499f0d751ded272d151"
server_key_iv: "7ca76e6905305b4d669803f8d71e0424"
server_key_version: "encryptionkey@0"
I20260812 11:29:51.687731 19124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:51.708098 19124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:51.712119 19124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:51.712393 19124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:51.731063 19124 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.252:32985
I20260812 11:29:51.731098 19230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.252:32985 every 8 connection(s)
I20260812 11:29:51.732308 19124 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
I20260812 11:29:51.735363 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19124
I20260812 11:29:51.735580 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786533802845873-12731-0/minicluster-data/master-2/wal/instance
I20260812 11:29:51.736146 19231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:51.736174 12731 external_mini_cluster.cc:1428] Setting key fe5cb40f8d774eb3dafd7bf4f858fb7b
I20260812 11:29:51.746215 19008 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 4
I20260812 11:29:51.745705 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:29:51.763340 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.737961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.12.110.252:32985)
0812 11:29:51.738796 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.738807 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.738848 (+ 41us) server_negotiation.cc:407] Connection header received
0812 11:29:51.740664 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.740685 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.741014 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.741303 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.743245 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.744524 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.746381 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.747517 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.748493 (+ 976us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.754006 (+ 5513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.754020 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.754041 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.754098 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.758759 (+ 4661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.759568 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.759577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.759587 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.759810 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.760331 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.760339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.760341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.760828 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.761210 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.761638 (+ 428us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.762083 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":648,"thread_start_us":370,"threads_started":1}
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:51.766026 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.746923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43162)
0812 11:29:51.747193 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.747213 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.747335 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.748226 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.748234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.748810 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.749062 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.749077 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.751252 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.751260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.753179 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.753200 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.754682 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.754691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.754934 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.756485 (+ 1551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.756513 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.759049 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.763242 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.763247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.763251 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.763829 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.764293 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.764298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.764302 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.764466 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.765088 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.765093 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.765229 (+ 136us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.765386 (+ 157us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.765662 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260812 11:29:51.766109 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.747108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43162 (local address 127.12.110.252:32985)
0812 11:29:51.747863 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.747868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.747892 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:29:51.747939 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.747943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.748008 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.748141 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.749310 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.751027 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.753441 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.754443 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.755222 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.759268 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.759275 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.759281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.759330 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.763109 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.763958 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.763965 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.763970 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.764146 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.764596 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.764601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.764604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.764933 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.765090 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.765476 (+ 386us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.765659 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":613,"thread_start_us":462,"threads_started":1}
I20260812 11:29:51.772490 19008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210: Bootstrap starting.
I20260812 11:29:51.774916 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.749353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:46289 (local address 127.0.0.1:45774)
0812 11:29:51.755986 (+ 6633us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.756045 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.756209 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.757046 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.757054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.757914 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.758285 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.758300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.759681 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.759686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.761552 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.761570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.762703 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.762709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.762917 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.763688 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.763724 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.767201 (+ 3477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.771618 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.771630 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.771637 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.772180 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.772649 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.772654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.772658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.772868 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.773573 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.773637 (+ 64us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.773867 (+ 230us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.774111 (+ 244us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.774415 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6408,"thread_start_us":6226,"threads_started":1}
I20260812 11:29:51.774956 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.749384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.12.110.254:46289)
0812 11:29:51.756680 (+ 7296us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.756688 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.756708 (+ 20us) server_negotiation.cc:407] Connection header received
0812 11:29:51.756754 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.756758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.756852 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.756977 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.758482 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.759512 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.761757 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.762440 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.762618 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.767486 (+ 4868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.767497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.767502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.767539 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.771476 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.772354 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.772357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.772359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.772480 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.773012 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.773017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.773021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.773452 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.773579 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.774250 (+ 671us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.774492 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7132,"thread_start_us":6954,"threads_started":1}
I20260812 11:29:51.776353 19008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210: Neither blocks nor log segments found. Creating new log.
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:51.777400 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:51.777602 19008 log.cc:824] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:51.781224 19008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210: No bootstrap required, opened a new log
I20260812 11:29:51.785413 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.767121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.12.110.254:46289)
0812 11:29:51.767724 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.767729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.767753 (+ 24us) server_negotiation.cc:407] Connection header received
0812 11:29:51.767801 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.767810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.767889 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.768002 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.769729 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.770689 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.772079 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.772808 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.772945 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.777811 (+ 4866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.777818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.777822 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.777861 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.781773 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.782489 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.782496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.782499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.782670 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.783127 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.783132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.783135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.783537 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.783782 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.784884 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.785112 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"spinlock_wait_cycles":23296,"thread_start_us":234,"threads_started":1}
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:29:51.791290 19008 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.791775 19008 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40d73cee3f734bb99862d846e96c5210, State: Initialized, Role: FOLLOWER
I20260812 11:29:51.792299 19008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.793555 19248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:51.794137 19248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:51.794950 19008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:51.795389 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.778222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:46775 (local address 127.0.0.1:49194)
0812 11:29:51.778471 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.778489 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.778696 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.779279 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.779286 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.779712 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.779930 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.779942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.780904 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.780910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.781848 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.781859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.782929 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.782935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.783131 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.783697 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.783719 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.786254 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.791889 (+ 5635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.791898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.791902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.792698 (+ 796us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.793321 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.793326 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.793329 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.793552 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.794311 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.794317 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.794504 (+ 187us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.794703 (+ 199us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.795025 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260812 11:29:51.795637 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.778264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.12.110.253:46775)
0812 11:29:51.778482 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.778490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.778592 (+ 102us) server_negotiation.cc:407] Connection header received
0812 11:29:51.778867 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.778875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.778975 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.779052 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.780084 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.780746 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.782085 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.782751 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.782896 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.786447 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.786454 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.786459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.786497 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.791657 (+ 5160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.792910 (+ 1253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.792919 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.792922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.793158 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.793695 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.793700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.793703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.794069 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.794238 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.794845 (+ 607us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.795124 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:29:51.797194 19231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:29:51.809924 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.787762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.12.110.253:46775)
0812 11:29:51.788375 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.788382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.788410 (+ 28us) server_negotiation.cc:407] Connection header received
0812 11:29:51.788460 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.788469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.788547 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.788666 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.791348 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.792642 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.794821 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.795552 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.795975 (+ 423us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.801948 (+ 5973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.801955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.801959 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.801998 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.805841 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.806737 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.806744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.806750 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.806909 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.807390 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.807395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.807398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.807815 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.808087 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.809242 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.809477 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":274,"threads_started":1}
I20260812 11:29:51.814951 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.798380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43164 (local address 127.12.110.252:32985)
0812 11:29:51.798681 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.798690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.799094 (+ 404us) server_negotiation.cc:407] Connection header received
0812 11:29:51.799262 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.799270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.799374 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.799500 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.800430 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.801843 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.803597 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.804370 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.804473 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.808315 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.808320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.808325 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.808358 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.812074 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.812861 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.812866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.812869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.813011 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.813403 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.813407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.813411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.813765 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.813892 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.814387 (+ 495us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.814625 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260812 11:29:51.815749 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.798465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43164)
0812 11:29:51.798971 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.798993 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.799135 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.799545 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.799550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.800054 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.800267 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.800281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.801996 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.802002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.803440 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.803457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.804693 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.804701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.804892 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.805484 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.805507 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.808144 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.812218 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.812223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.812227 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.812746 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.813148 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.813153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.813157 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.813302 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.813951 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.813955 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.814134 (+ 179us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.815134 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.815415 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373}
I20260812 11:29:51.815819 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:51.819476 18972 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "40d73cee3f734bb99862d846e96c5210"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45782
I20260812 11:29:51.819860 18972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 11:29:51.820055 18972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:51.821072 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe: Bootstrap starting.
I20260812 11:29:51.811905 18904 master_runner.cc:186] Error getting master registration for 127.12.110.253:46775: OK, instance_id { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" instance_seqno: 1786534191157724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:29:51.821960 18904 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:29:51.823316 18972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.825023 19231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:51.825093 18972 leader_election.cc:288] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [CANDIDATE]: Term 1 election: Requested vote from peers a9a2f320947d42fa843031bbaa5c9996 (127.12.110.253:46775), c32930a11ee241b885a35e96eb819ffe (127.12.110.252:32985)
W20260812 11:29:51.826130 19261 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:29:51.826398 19261 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:29:51.827396 19231 log.cc:824] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:51.831648 19231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe: No bootstrap required, opened a new log
W20260812 11:29:51.840687 18941 tablet.cc:2369] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:29:51.852303 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.830723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45786 (local address 127.12.110.254:46289)
0812 11:29:51.831273 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.831281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.831566 (+ 285us) server_negotiation.cc:407] Connection header received
0812 11:29:51.831847 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.831853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.831948 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.832037 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.834775 (+ 2738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.836152 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.838549 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.839402 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.839507 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.844303 (+ 4796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.844312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.844323 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.844367 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.848370 (+ 4003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.849661 (+ 1291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.849668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.849672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.849823 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.850311 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.850317 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.850321 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.850746 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.851041 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.851662 (+ 621us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.851894 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371}
I20260812 11:29:51.856796 19231 raft_consensus.cc:348] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.857709 19231 raft_consensus.cc:729] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c32930a11ee241b885a35e96eb819ffe, State: Initialized, Role: FOLLOWER
I20260812 11:29:51.858657 19231 consensus_queue.cc:260] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.863547 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.832129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.12.110.252:32985)
0812 11:29:51.832285 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.832291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.843234 (+ 10943us) server_negotiation.cc:407] Connection header received
0812 11:29:51.843735 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.843742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.843823 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.843896 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.845779 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.846965 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.848896 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.849706 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.849816 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.854921 (+ 5105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.854931 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.854936 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.854974 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.858925 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.860287 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.860293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.860297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.860455 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.861178 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.861184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.861187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.861534 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.861797 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.862879 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.863125 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:29:51.867265 19275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:51.869408 19231 sys_catalog.cc:564] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:51.869477 19189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40d73cee3f734bb99862d846e96c5210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c32930a11ee241b885a35e96eb819ffe"
I20260812 11:29:51.869998 19189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:51.870891 19275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:51.872344 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.840402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.12.110.253:46775)
0812 11:29:51.840585 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.840591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.854956 (+ 14365us) server_negotiation.cc:407] Connection header received
0812 11:29:51.855259 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.855266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.855348 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.855421 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.858014 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.859164 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.860968 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.861636 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.861729 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.864918 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.864925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.864929 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.864971 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.868689 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.869584 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.869591 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.869597 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.869745 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.870347 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.870353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.870356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.870766 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.871042 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.871589 (+ 547us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.871816 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:29:51.874349 19189 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d73cee3f734bb99862d846e96c5210 in term 1.
I20260812 11:29:51.874979 18934 leader_election.cc:302] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40d73cee3f734bb99862d846e96c5210, c32930a11ee241b885a35e96eb819ffe; no voters:
I20260812 11:29:51.875461 19248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:51.876185 19248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [term 1 LEADER]: Becoming Leader. State: Replica: 40d73cee3f734bb99862d846e96c5210, State: Running, Role: LEADER
I20260812 11:29:51.876757 19248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.884850 19280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d73cee3f734bb99862d846e96c5210. Latest consensus state: current_term: 1 leader_uuid: "40d73cee3f734bb99862d846e96c5210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:51.885386 19280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:51.886442 19285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260812 11:29:51.889290 19114 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } attempt: 5
I20260812 11:29:51.889523 19285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 12 11:29:51 dist-test-slave-h6n0 krb5kdc[18883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534191, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:29:51.909461 19189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [term 1 FOLLOWER]: Refusing update from remote peer 40d73cee3f734bb99862d846e96c5210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:51.912070 19248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:29:51.912477 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.889803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:32985 (local address 127.0.0.1:43186)
0812 11:29:51.890140 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:51.890162 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:51.890272 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:51.891269 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:51.891277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:51.891828 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:51.892058 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.892073 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.894456 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.894465 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.896369 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:51.896389 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.898217 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.898227 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:51.898738 (+ 511us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.902269 (+ 3531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:51.902293 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:51.905120 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:51.909328 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.909334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.909337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.909844 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.910309 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:51.910314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:51.910317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:51.910492 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:51.911264 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:51.911270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:51.911429 (+ 159us) client_negotiation.cc:769] Sending connection context
0812 11:29:51.911581 (+ 152us) client_negotiation.cc:240] Negotiation successful
0812 11:29:51.911880 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260812 11:29:51.913578 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:51.890487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43186 (local address 127.12.110.252:32985)
0812 11:29:51.890684 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:51.890689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:51.890707 (+ 18us) server_negotiation.cc:407] Connection header received
0812 11:29:51.890759 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:51.890764 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:51.890835 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:51.890954 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:51.892338 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.893465 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.896634 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:51.897685 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:51.897836 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:51.905301 (+ 7465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:51.905309 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:51.905315 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:51.905374 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:51.909174 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.909995 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.910013 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.910016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.910168 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:51.910656 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:51.910661 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:51.910664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:51.911049 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:51.911197 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:51.912826 (+ 1629us) server_negotiation.cc:299] Negotiation successful
0812 11:29:51.913112 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260812 11:29:51.920847 19114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996: Bootstrap starting.
I20260812 11:29:51.925921 19114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:51.927593 19114 log.cc:824] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:51.936411 19275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d73cee3f734bb99862d846e96c5210. Latest consensus state: current_term: 1 leader_uuid: "40d73cee3f734bb99862d846e96c5210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:51.936735 19275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:51.938483 19290 mvcc.cc:205] Tried to move back new op lower bound from 7317644050043797504 to 7317644049932382208. Current Snapshot: MvccSnapshot[applied={T|T < 7317644050043797504}]
I20260812 11:29:51.941556 19114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996: No bootstrap required, opened a new log
I20260812 11:29:51.942438 19248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40d73cee3f734bb99862d846e96c5210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:51.943532 19248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:51.945813 19285 catalog_manager.cc:1347] Generated new cluster ID: 423998a54b8147e68f302df66d5fa9b4
I20260812 11:29:51.945926 19285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:51.955233 19275 sys_catalog.cc:455] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40d73cee3f734bb99862d846e96c5210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:51.955906 19275 sys_catalog.cc:458] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:51.960469 19114 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.961011 19114 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a2f320947d42fa843031bbaa5c9996, State: Initialized, Role: FOLLOWER
I20260812 11:29:51.961982 19114 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } }
I20260812 11:29:51.974484 19114 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:51.976295 19285 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:51.972823 19306 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:51.976486 19306 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:51.976707 19081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "40d73cee3f734bb99862d846e96c5210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9a2f320947d42fa843031bbaa5c9996"
I20260812 11:29:51.977504 19079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:51.978236 19285 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:51.981714 19079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [term 1 FOLLOWER]: Refusing update from remote peer 40d73cee3f734bb99862d846e96c5210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:51.983038 19307 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260812 11:29:51.993639 19081 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40d73cee3f734bb99862d846e96c5210 in term 1.
I20260812 11:29:51.998631 19285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210: Generated new TSK 0
I20260812 11:29:51.999624 19285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:51.999558 19248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:29:51.993922 19124 master_runner.cc:186] Error getting master registration for 127.12.110.253:46775: OK, instance_id { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" instance_seqno: 1786534191157724 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:29:52.003993 19124 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:29:52.015193 19306 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40d73cee3f734bb99862d846e96c5210. Latest consensus state: current_term: 1 leader_uuid: "40d73cee3f734bb99862d846e96c5210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:52.015537 19306 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:52.015116 19323 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "40d73cee3f734bb99862d846e96c5210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:52.015730 19323 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:52.029932 19328 catalog_manager.cc:1259] Loaded cluster ID: 423998a54b8147e68f302df66d5fa9b4
I20260812 11:29:52.030097 19328 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996: loading cluster ID for follower catalog manager: success
I20260812 11:29:52.036701 19328 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996: acquiring CA information for follower catalog manager: success
I20260812 11:29:52.039242 19328 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9a2f320947d42fa843031bbaa5c9996: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 11:29:52.123673 19244 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.109727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45798 (local address 127.12.110.254:46289)
0812 11:29:52.109918 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.109924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.120281 (+ 10357us) server_negotiation.cc:407] Connection header received
0812 11:29:52.121286 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.121293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.121383 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.121463 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.123365 (+ 1902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45798: BlockingRecv error: recv got EOF from 127.0.0.1:45798 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260812 11:29:52.123687 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.109577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.12.110.252:32985)
0812 11:29:52.109766 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.109772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.121619 (+ 11847us) server_negotiation.cc:407] Connection header received
0812 11:29:52.122078 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.122085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.122190 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.122261 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.123409 (+ 1148us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43188: BlockingRecv error: recv got EOF from 127.0.0.1:43188 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 11:29:52.126012 19246 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.115837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.12.110.253:46775)
0812 11:29:52.116030 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.116036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.124412 (+ 8376us) server_negotiation.cc:407] Connection header received
0812 11:29:52.124679 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.124685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.124760 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.124828 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.125740 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49226: BlockingRecv error: recv got EOF from 127.0.0.1:49226 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:29:52.138082 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.130309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.12.110.253:46775)
0812 11:29:52.130488 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.130494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.131000 (+ 506us) server_negotiation.cc:407] Connection header received
0812 11:29:52.131192 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.131197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.131402 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.131570 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.132029 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.133176 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.135221 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.135953 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.136086 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.136636 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.137381 (+ 745us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.137528 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.137662 (+ 134us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.137746 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 11:29:52.151739 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.145259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43204 (local address 127.12.110.252:32985)
0812 11:29:52.145430 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.145436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.150195 (+ 4759us) server_negotiation.cc:407] Connection header received
0812 11:29:52.150390 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.150395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.150575 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.150684 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.151489 (+ 805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43204: BlockingRecv error: recv got EOF from 127.0.0.1:43204 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260812 11:29:52.167220 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.155926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.12.110.253:46775)
0812 11:29:52.156083 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.156089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.156573 (+ 484us) server_negotiation.cc:407] Connection header received
0812 11:29:52.156768 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.156773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.156852 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.156949 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.157395 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.158579 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.160590 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.161330 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.161480 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.161942 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.162092 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.162183 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.162354 (+ 171us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.162430 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260812 11:29:52.199019 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.197112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.12.110.252:32985)
0812 11:29:52.197315 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.197324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.197430 (+ 106us) server_negotiation.cc:407] Connection header received
0812 11:29:52.197725 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.197731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.197824 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.197952 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.198784 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43216: BlockingRecv error: recv got EOF from 127.0.0.1:43216 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:29:52.223085 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.214429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.12.110.253:46775)
0812 11:29:52.214675 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.214681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.215233 (+ 552us) server_negotiation.cc:407] Connection header received
0812 11:29:52.215556 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.215564 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.215671 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.215753 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.216438 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.217632 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.220262 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.221149 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.221356 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.222012 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.222212 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.222324 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.222582 (+ 258us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.222746 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260812 11:29:52.237282 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.230637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43220 (local address 127.12.110.252:32985)
0812 11:29:52.230829 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.230836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.235528 (+ 4692us) server_negotiation.cc:407] Connection header received
0812 11:29:52.235734 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.235740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.235842 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.236081 (+ 239us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.237033 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43220: BlockingRecv error: recv got EOF from 127.0.0.1:43220 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:29:52.252909 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.245349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.12.110.253:46775)
0812 11:29:52.245585 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.245591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.246151 (+ 560us) server_negotiation.cc:407] Connection header received
0812 11:29:52.246388 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.246398 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.246522 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.246676 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.247179 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.248674 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.250698 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.251389 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.251541 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.252024 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.252194 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.252299 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.252464 (+ 165us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.252569 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260812 11:29:52.275883 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.268308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.12.110.252:32985)
0812 11:29:52.268570 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.268578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.269157 (+ 579us) server_negotiation.cc:407] Connection header received
0812 11:29:52.269363 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.269370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.269467 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.269601 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.270565 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43230: BlockingRecv error: recv got EOF from 127.0.0.1:43230 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
I20260812 11:29:52.291486 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.284115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49278 (local address 127.12.110.253:46775)
0812 11:29:52.284343 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.284350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.284950 (+ 600us) server_negotiation.cc:407] Connection header received
0812 11:29:52.285288 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.285295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.285427 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.285556 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.286039 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.287210 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.289118 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.289838 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.289973 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.290517 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.290698 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.290795 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.290978 (+ 183us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.291077 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 11:29:52.305601 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.298958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43236 (local address 127.12.110.252:32985)
0812 11:29:52.299141 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.299147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.303931 (+ 4784us) server_negotiation.cc:407] Connection header received
0812 11:29:52.304183 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.304189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.304293 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.304409 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.305347 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43236: BlockingRecv error: recv got EOF from 127.0.0.1:43236 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260812 11:29:52.335168 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.326801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49292 (local address 127.12.110.253:46775)
0812 11:29:52.327021 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.327027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.327547 (+ 520us) server_negotiation.cc:407] Connection header received
0812 11:29:52.327760 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.327766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.327853 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.327956 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.328517 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.329705 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.331986 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.332713 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.332817 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.333635 (+ 818us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.333920 (+ 285us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.334041 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.334248 (+ 207us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.334376 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 11:29:52.345247 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.342859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.12.110.252:32985)
0812 11:29:52.343067 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.343072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.343662 (+ 590us) server_negotiation.cc:407] Connection header received
0812 11:29:52.343852 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.343858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.343958 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.344074 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.345005 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43250: BlockingRecv error: recv got EOF from 127.0.0.1:43250 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:29:52.389601 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.381915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.12.110.253:46775)
0812 11:29:52.382212 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.382219 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.383104 (+ 885us) server_negotiation.cc:407] Connection header received
0812 11:29:52.383319 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.383326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.383433 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.383560 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.384133 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.385342 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.387347 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.388011 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.388177 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.388689 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.388862 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.388967 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.389159 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.389266 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260812 11:29:52.406713 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.404497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43264 (local address 127.12.110.252:32985)
0812 11:29:52.404669 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.404674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.404856 (+ 182us) server_negotiation.cc:407] Connection header received
0812 11:29:52.405180 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.405188 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.405304 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.405407 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.406445 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43264: BlockingRecv error: recv got EOF from 127.0.0.1:43264 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260812 11:29:52.481232 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.474600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49314 (local address 127.12.110.253:46775)
0812 11:29:52.474896 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.474903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.475169 (+ 266us) server_negotiation.cc:407] Connection header received
0812 11:29:52.475341 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.475353 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.475448 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.475560 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.476067 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.477295 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.479178 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.479818 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.479958 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.480398 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.480578 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.480661 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.480806 (+ 145us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.480883 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260812 11:29:52.488775 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.486786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.12.110.252:32985)
0812 11:29:52.486973 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.486979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.487333 (+ 354us) server_negotiation.cc:407] Connection header received
0812 11:29:52.487472 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.487478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.487567 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.487690 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.488545 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43280: BlockingRecv error: recv got EOF from 127.0.0.1:43280 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260812 11:29:52.565160 19248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "40d73cee3f734bb99862d846e96c5210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40d73cee3f734bb99862d846e96c5210" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 46289 } } peers { permanent_uuid: "a9a2f320947d42fa843031bbaa5c9996" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 46775 } } peers { permanent_uuid: "c32930a11ee241b885a35e96eb819ffe" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 32985 } } }
I20260812 11:29:52.565604 19248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40d73cee3f734bb99862d846e96c5210 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:52.625859 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.620191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49328 (local address 127.12.110.253:46775)
0812 11:29:52.620383 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.620389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.620745 (+ 356us) server_negotiation.cc:407] Connection header received
0812 11:29:52.620918 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.620924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.621009 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.621124 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.621570 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.622690 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.624197 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.624688 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.624786 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.625240 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.625338 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.625408 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.625588 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.625650 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260812 11:29:52.633846 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.631845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.12.110.252:32985)
0812 11:29:52.632132 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.632139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.632425 (+ 286us) server_negotiation.cc:407] Connection header received
0812 11:29:52.632552 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.632556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.632628 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.632710 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.633561 (+ 851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43290: BlockingRecv error: recv got EOF from 127.0.0.1:43290 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260812 11:29:52.900629 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:52.894513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.12.110.253:46775)
0812 11:29:52.894758 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.894763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.894988 (+ 225us) server_negotiation.cc:407] Connection header received
0812 11:29:52.895215 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.895220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.895288 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.895369 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:52.895864 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.896937 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.898678 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:52.899096 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:52.899271 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:52.899635 (+ 364us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:52.899738 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:52.899820 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:52.900215 (+ 395us) server_negotiation.cc:299] Negotiation successful
0812 11:29:52.900281 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260812 11:29:52.909232 19239 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:52.907154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43294 (local address 127.12.110.252:32985)
0812 11:29:52.907369 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:52.907375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:52.907761 (+ 386us) server_negotiation.cc:407] Connection header received
0812 11:29:52.907977 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:52.907982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:52.908116 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:52.908226 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:52.909032 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43294: BlockingRecv error: recv got EOF from 127.0.0.1:43294 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260812 11:29:52.987452 19307 catalog_manager.cc:1259] Loaded cluster ID: 423998a54b8147e68f302df66d5fa9b4
I20260812 11:29:52.987622 19307 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe: loading cluster ID for follower catalog manager: success
I20260812 11:29:52.990245 19307 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe: acquiring CA information for follower catalog manager: success
I20260812 11:29:52.992208 19307 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c32930a11ee241b885a35e96eb819ffe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:29:53.430353 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:53.424844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.12.110.253:46775)
0812 11:29:53.425376 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:53.425383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:53.425726 (+ 343us) server_negotiation.cc:407] Connection header received
0812 11:29:53.425970 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:53.425974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:53.426045 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:53.426156 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:53.426616 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.427317 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.428669 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.429078 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.429218 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:53.429614 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:53.429725 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:53.429829 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:53.429991 (+ 162us) server_negotiation.cc:299] Negotiation successful
0812 11:29:53.430067 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
I20260812 11:29:53.442101 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:53.436174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.12.110.252:32985)
0812 11:29:53.436669 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:53.436673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:53.436741 (+ 68us) server_negotiation.cc:407] Connection header received
0812 11:29:53.436937 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:53.436942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:53.437139 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:53.437250 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:53.437707 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.438417 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.439822 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.440277 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.440398 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:53.440897 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:53.441406 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:53.441550 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:53.441659 (+ 109us) server_negotiation.cc:299] Negotiation successful
0812 11:29:53.441723 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":247,"threads_started":1}
I20260812 11:29:53.463503 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:53.449332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.12.110.253:46775)
0812 11:29:53.449886 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:53.449895 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:53.456289 (+ 6394us) server_negotiation.cc:407] Connection header received
0812 11:29:53.457053 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:53.457058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:53.457126 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:53.457180 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:53.458169 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.459249 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.461522 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.462087 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.462210 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:53.462734 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:53.462869 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:53.462964 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:53.463159 (+ 195us) server_negotiation.cc:299] Negotiation successful
0812 11:29:53.463242 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260812 11:29:53.464799 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:53.449559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.12.110.254:46289)
0812 11:29:53.456454 (+ 6895us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:53.456463 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:53.457304 (+ 841us) server_negotiation.cc:407] Connection header received
0812 11:29:53.457822 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:53.457846 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:53.458072 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:53.458285 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:53.458839 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.459804 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.461766 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.462383 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.462473 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:53.463271 (+ 798us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:53.463694 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:53.463795 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:53.464403 (+ 608us) server_negotiation.cc:299] Negotiation successful
0812 11:29:53.464483 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6748,"thread_start_us":6833,"threads_started":1}
I20260812 11:29:53.465351 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:53.449263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.12.110.252:32985)
0812 11:29:53.449424 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:53.449431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:53.457240 (+ 7809us) server_negotiation.cc:407] Connection header received
0812 11:29:53.457393 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:53.457399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:53.457483 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:53.457586 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:53.458293 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.459316 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.463080 (+ 3764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:53.463650 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:53.463779 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:53.464240 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:53.464364 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:53.464456 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:53.464598 (+ 142us) server_negotiation.cc:299] Negotiation successful
0812 11:29:53.464674 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260812 11:29:53.473990 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 18904
I20260812 11:29:53.475417 18997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:53.588447 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 18904
W20260812 11:29:53.620085 19049 connection.cc:535] server connection from 127.0.0.1:49190 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260812 11:29:53.620544 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19019
I20260812 11:29:53.621742 19106 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:53.732977 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19019
I20260812 11:29:53.766499 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19124
I20260812 11:29:53.767645 19221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:53.874348 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19124
2026-08-12T11:29:53Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5466 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 12 11:29:54 dist-test-slave-h6n0 krb5kdc[19562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 12 11:29:54 dist-test-slave-h6n0 krb5kdc[19562](info): set up 2 sockets
Aug 12 11:29:54 dist-test-slave-h6n0 krb5kdc[19562](info): commencing operation
krb5kdc: starting...
W20260812 11:29:55.285614 12731 mini_kdc.cc:121] Time spent starting KDC: real 1.363s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/test-user.keytab.
Aug 12 11:29:55 dist-test-slave-h6n0 krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-12T11:29:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-12T11:29:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.12.110.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
Entry for principal kudu/127.12.110.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab.
I20260812 11:29:55.644270 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36323
--webserver_interface=127.12.110.254
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36323,127.12.110.253:41751,127.12.110.252:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260812 11:29:55.988831 19590 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:29:55.989224 19590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:55.989356 19590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:55.989497 19590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:55.996629 19590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:55.996757 19590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:55.996873 19590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:55.996963 19590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:55.997056 19590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:56.006646 19590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36323,127.12.110.253:41751,127.12.110.252:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.254.keytab
--principal=kudu/127.12.110.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.254:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.12.110.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:56.007933 19590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:56.009268 19590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:56.020835 19600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:56.020927 19601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:56.020895 19603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:56.021656 19590 server_base.cc:1034] running on GCE node
I20260812 11:29:56.022433 19590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:56.023870 19590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:56.025148 19590 hybrid_clock.cc:648] HybridClock initialized: now 1786534196025080 us; error 41 us; skew 500 ppm
I20260812 11:29:56.025524 19590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:56 dist-test-slave-h6n0 krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:56.033390 19590 init.cc:377] Logged in from keytab as kudu/127.12.110.254@KRBTEST.COM (short username kudu)
I20260812 11:29:56.035115 19590 webserver.cc:460] Webserver started at http://127.12.110.254:33653/ using document root <none> and password file <none>
I20260812 11:29:56.035835 19590 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:56.035952 19590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:56.036302 19590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:56.039367 19590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data/instance:
uuid: "3d1c86b8af9449fcb6ae48deee769396"
format_stamp: "Formatted at 2026-08-12 11:29:56 on dist-test-slave-h6n0"
server_key: "5dff4680787e7f70496d5d3eaf4b7b46"
server_key_iv: "e9de7bd408127160de535d3341ddb8aa"
server_key_version: "encryptionkey@0"
I20260812 11:29:56.040184 19590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/wal/instance:
uuid: "3d1c86b8af9449fcb6ae48deee769396"
format_stamp: "Formatted at 2026-08-12 11:29:56 on dist-test-slave-h6n0"
server_key: "5dff4680787e7f70496d5d3eaf4b7b46"
server_key_iv: "e9de7bd408127160de535d3341ddb8aa"
server_key_version: "encryptionkey@0"
I20260812 11:29:56.045280 19590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260812 11:29:56.048734 19616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:56.050067 19590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260812 11:29:56.050235 19590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/wal
uuid: "3d1c86b8af9449fcb6ae48deee769396"
format_stamp: "Formatted at 2026-08-12 11:29:56 on dist-test-slave-h6n0"
server_key: "5dff4680787e7f70496d5d3eaf4b7b46"
server_key_iv: "e9de7bd408127160de535d3341ddb8aa"
server_key_version: "encryptionkey@0"
I20260812 11:29:56.050453 19590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:56.071872 19590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:56.076453 19590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:56.076695 19590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:56.096746 19590 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.254:36323
I20260812 11:29:56.096765 19707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.254:36323 every 8 connection(s)
I20260812 11:29:56.098085 19590 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/data/info.pb
I20260812 11:29:56.099311 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19590
I20260812 11:29:56.099520 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-0/wal/instance
I20260812 11:29:56.100093 12731 external_mini_cluster.cc:1428] Setting key 77d56caa5254555a634777148561516c
I20260812 11:29:56.102615 19708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:56.112555 19708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:56 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.12.110.253@KRBTEST.COM; defaulting to no policy
I20260812 11:29:56.141077 19716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.116672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.12.110.254:36323)
0812 11:29:56.117606 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:56.117621 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:56.118181 (+ 560us) server_negotiation.cc:407] Connection header received
0812 11:29:56.121173 (+ 2992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:56.121179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:56.121513 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:56.121672 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:56.123463 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.124676 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.126955 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.127507 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.127643 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.132610 (+ 4967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:56.132620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:56.132637 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:56.132674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:56.136406 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.137194 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.137208 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.137210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.137359 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.137775 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.137782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.137790 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.138344 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:56.138603 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:56.139121 (+ 518us) server_negotiation.cc:299] Negotiation successful
0812 11:29:56.139546 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":671,"spinlock_wait_cycles":48000,"thread_start_us":411,"threads_started":1}
I20260812 11:29:56.141072 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.116672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36323 (local address 127.0.0.1:49198)
0812 11:29:56.117993 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.118044 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:56.120791 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:56.121762 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:56.121772 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:56.122626 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:56.123213 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.123245 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.124848 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.124857 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.126808 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.126826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.127762 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.127772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.127994 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.128767 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:56.128801 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:56.132268 (+ 3467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:56.136565 (+ 4297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.136580 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.136593 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.137029 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.137486 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.137490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.137494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.137656 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.138482 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:56.138490 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:56.138742 (+ 252us) client_negotiation.cc:769] Sending connection context
0812 11:29:56.138988 (+ 246us) client_negotiation.cc:240] Negotiation successful
0812 11:29:56.139675 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":176,"thread_start_us":831,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.253@KRBTEST.COM" created.
I20260812 11:29:56.144246 19708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:56.145500 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.144977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36480)
0812 11:29:56.145204 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.145348 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260812 11:29:56.145809 19620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
I20260812 11:29:56.147482 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.147037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36490)
0812 11:29:56.147211 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.147303 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:29:56.147703 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:41751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab.
Entry for principal kudu/127.12.110.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab.
I20260812 11:29:56.183388 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:41751
--webserver_interface=127.12.110.253
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab
--principal=kudu/127.12.110.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36323,127.12.110.253:41751,127.12.110.252:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:29:56.197063 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } attempt: 1
I20260812 11:29:56.198235 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.197751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36492)
0812 11:29:56.197972 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.198106 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260812 11:29:56.199327 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.198973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36494)
0812 11:29:56.199122 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.199202 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260812 11:29:56.199529 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:41751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
I20260812 11:29:56.267791 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } attempt: 2
I20260812 11:29:56.269081 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.268483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36504)
0812 11:29:56.268741 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.268836 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 11:29:56.270654 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.269882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36512)
0812 11:29:56.270357 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.270441 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192}
W20260812 11:29:56.270916 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:41751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
I20260812 11:29:56.362221 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } attempt: 3
I20260812 11:29:56.363440 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.362920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36520)
0812 11:29:56.363154 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.363269 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260812 11:29:56.364516 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.364095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36530)
0812 11:29:56.364245 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.364330 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260812 11:29:56.364719 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:41751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
I20260812 11:29:56.507990 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } attempt: 4
I20260812 11:29:56.509316 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.508702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36546)
0812 11:29:56.509041 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.509151 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260812 11:29:56.510401 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.509990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36556)
0812 11:29:56.510153 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.510250 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260812 11:29:56.510663 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.253:41751: Network error: Client connection negotiation failed: client connection to 127.12.110.253:41751: connect: Connection refused (error 111)
W20260812 11:29:56.530670 19721 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:29:56.530941 19721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:56.531004 19721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:56.531085 19721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:56.538079 19721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:56.538153 19721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:56.538208 19721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:56.538259 19721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:56.538313 19721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:56.547612 19721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36323,127.12.110.253:41751,127.12.110.252:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.253.keytab
--principal=kudu/127.12.110.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.253:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.12.110.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:56.548707 19721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:56.549862 19721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:56.561545 19727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:56.561498 19729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:56.561461 19736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:56.562834 19721 server_base.cc:1034] running on GCE node
I20260812 11:29:56.563530 19721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:56.564968 19721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:56.566255 19721 hybrid_clock.cc:648] HybridClock initialized: now 1786534196566191 us; error 60 us; skew 500 ppm
I20260812 11:29:56.566709 19721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:56 dist-test-slave-h6n0 krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:56.570132 19721 init.cc:377] Logged in from keytab as kudu/127.12.110.253@KRBTEST.COM (short username kudu)
I20260812 11:29:56.571975 19721 webserver.cc:460] Webserver started at http://127.12.110.253:33181/ using document root <none> and password file <none>
I20260812 11:29:56.572739 19721 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:56.572897 19721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:56.573249 19721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:56.576646 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data/instance:
uuid: "1484cc504ecc4b6fbdf761104796b406"
format_stamp: "Formatted at 2026-08-12 11:29:56 on dist-test-slave-h6n0"
server_key: "6d7173ca1dc48dd24f6db04e0366ca4a"
server_key_iv: "8908159e75bd540413ee3f755eba269a"
server_key_version: "encryptionkey@0"
I20260812 11:29:56.577498 19721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/wal/instance:
uuid: "1484cc504ecc4b6fbdf761104796b406"
format_stamp: "Formatted at 2026-08-12 11:29:56 on dist-test-slave-h6n0"
server_key: "6d7173ca1dc48dd24f6db04e0366ca4a"
server_key_iv: "8908159e75bd540413ee3f755eba269a"
server_key_version: "encryptionkey@0"
I20260812 11:29:56.583132 19721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260812 11:29:56.587862 19745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:56.589288 19721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260812 11:29:56.589514 19721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/wal
uuid: "1484cc504ecc4b6fbdf761104796b406"
format_stamp: "Formatted at 2026-08-12 11:29:56 on dist-test-slave-h6n0"
server_key: "6d7173ca1dc48dd24f6db04e0366ca4a"
server_key_iv: "8908159e75bd540413ee3f755eba269a"
server_key_version: "encryptionkey@0"
I20260812 11:29:56.589762 19721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:56.615142 19721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:56.619560 19721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:56.619850 19721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:56.640594 19721 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.253:41751
I20260812 11:29:56.640625 19826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.253:41751 every 8 connection(s)
I20260812 11:29:56.641902 19721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/data/info.pb
I20260812 11:29:56.645356 19830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:56.646901 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19721
I20260812 11:29:56.647111 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-1/wal/instance
I20260812 11:29:56.647569 12731 external_mini_cluster.cc:1428] Setting key 475b59e037eea7f865479a64294ce060
I20260812 11:29:56.655036 19830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:56 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:56.681322 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.661972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.12.110.254:36323)
0812 11:29:56.662578 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:56.662587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:56.663075 (+ 488us) server_negotiation.cc:407] Connection header received
0812 11:29:56.665286 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:56.665295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:56.665398 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:56.665531 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:56.667107 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.667879 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.670343 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.670865 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.670982 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.675421 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:56.675426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:56.675430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:56.675472 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:56.678171 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.678999 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.679005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.679007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.679133 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.679572 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.679584 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.679587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.679857 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:56.680021 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:56.680725 (+ 704us) server_negotiation.cc:299] Negotiation successful
0812 11:29:56.680941 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":264,"threads_started":1}
I20260812 11:29:56.682333 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.660420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36323 (local address 127.0.0.1:49202)
0812 11:29:56.662910 (+ 2490us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.662957 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:56.665009 (+ 2052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:56.665741 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:56.665749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:56.666273 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:56.666890 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.666908 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.668036 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.668044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.670164 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.670183 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.671107 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.671117 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.671443 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.672264 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:56.672298 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:56.675188 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:56.678321 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.678330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.678345 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.678862 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.679260 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.679265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.679268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.679448 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.679995 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:56.680003 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:56.680348 (+ 345us) client_negotiation.cc:769] Sending connection context
0812 11:29:56.680630 (+ 282us) client_negotiation.cc:240] Negotiation successful
0812 11:29:56.681105 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2215,"thread_start_us":1988,"threads_started":1}
I20260812 11:29:56.684158 19830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.12.110.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.12.110.252@KRBTEST.COM" created.
Aug 12 11:29:56 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:29:56.701004 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.685077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36572)
0812 11:29:56.685237 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.685263 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:56.685380 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:56.686173 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:56.686178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:56.686725 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:56.686879 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.686888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.688284 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.688289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.689585 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.689603 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.691049 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.691058 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.691312 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.692032 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:56.692060 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:56.694779 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:56.697971 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.697975 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.697979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.698331 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.698764 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.698769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.698773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.699118 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.699955 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:56.699963 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:56.700116 (+ 153us) client_negotiation.cc:769] Sending connection context
0812 11:29:56.700275 (+ 159us) client_negotiation.cc:240] Negotiation successful
0812 11:29:56.700590 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260812 11:29:56.701128 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.685164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.12.110.253:41751)
0812 11:29:56.685760 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:56.685768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:56.685800 (+ 32us) server_negotiation.cc:407] Connection header received
0812 11:29:56.685853 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:56.685859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:56.686020 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:56.686175 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:56.687064 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.688142 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.689893 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.690814 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.690984 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.695018 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:56.695030 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:56.695042 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:56.695071 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:56.697817 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.698468 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.698482 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.698484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.698650 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.699418 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.699427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.699431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.699816 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:56.700065 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:56.700420 (+ 355us) server_negotiation.cc:299] Negotiation successful
0812 11:29:56.700713 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":433,"thread_start_us":285,"threads_started":1}
I20260812 11:29:56.703694 19830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:56.704885 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.704282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52394)
0812 11:29:56.704527 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.704671 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260812 11:29:56.705178 19748 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
I20260812 11:29:56.706732 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.706343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52400)
0812 11:29:56.706500 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.706637 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260812 11:29:56.707036 19830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.12.110.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab.
Entry for principal kudu/127.12.110.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab.
I20260812 11:29:56.729291 12731 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
/tmp/dist-test-tasklUqj9l/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:40031
--webserver_interface=127.12.110.252
--webserver_port=0
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab
--principal=kudu/127.12.110.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.12.110.212:35463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.12.110.254:36323,127.12.110.253:41751,127.12.110.252:40031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260812 11:29:56.756390 19830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 1
I20260812 11:29:56.757618 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.757159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52408)
0812 11:29:56.757355 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.757466 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260812 11:29:56.758716 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.758285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52424)
0812 11:29:56.758461 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.758595 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260812 11:29:56.758931 19830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
I20260812 11:29:56.809943 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } attempt: 5
Aug 12 11:29:56 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:29:56.825055 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.810824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36584)
0812 11:29:56.811077 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.811100 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:56.811239 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:56.811660 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:56.811663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:56.812039 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:56.812217 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.812224 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.813636 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.813640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.814519 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:56.814591 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.815606 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.815610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:56.815776 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.816448 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:56.816467 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:56.818569 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:56.822479 (+ 3910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.822483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.822485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.822974 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.823385 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:56.823388 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:56.823391 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:56.823533 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:56.824179 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:56.824182 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:56.824311 (+ 129us) client_negotiation.cc:769] Sending connection context
0812 11:29:56.824422 (+ 111us) client_negotiation.cc:240] Negotiation successful
0812 11:29:56.824777 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260812 11:29:56.825116 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.811085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.12.110.253:41751)
0812 11:29:56.811316 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:56.811323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:56.811345 (+ 22us) server_negotiation.cc:407] Connection header received
0812 11:29:56.811425 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:56.811430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:56.811511 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:56.811625 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:56.812377 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.813491 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.814739 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:56.815448 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:56.815568 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:56.818782 (+ 3214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:56.818790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:56.818798 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:56.818844 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:56.822351 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.823099 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.823105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.823109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.823255 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:56.823684 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:56.823689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:56.823693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:56.824048 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:56.824202 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:56.824596 (+ 394us) server_negotiation.cc:299] Negotiation successful
0812 11:29:56.824817 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260812 11:29:56.826290 19708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:56.827157 19830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 2
I20260812 11:29:56.827421 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.826970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52440)
0812 11:29:56.827162 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.827255 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260812 11:29:56.828089 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.827644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52448)
0812 11:29:56.827822 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.827928 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260812 11:29:56.828668 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.828268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52462)
0812 11:29:56.828456 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.828543 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260812 11:29:56.829114 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.828713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52478)
0812 11:29:56.828898 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.828988 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 11:29:56.829316 19830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
W20260812 11:29:56.829319 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
I20260812 11:29:56.880589 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 1
I20260812 11:29:56.881876 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.881301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52494)
0812 11:29:56.881544 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.881664 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260812 11:29:56.883081 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.882662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52498)
0812 11:29:56.882841 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.882937 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 11:29:56.883337 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
I20260812 11:29:56.920579 19830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 3
I20260812 11:29:56.922228 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.921617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52500)
0812 11:29:56.921870 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.921998 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260812 11:29:56.923959 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.923348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52510)
0812 11:29:56.923667 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.923792 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260812 11:29:56.924270 19830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
I20260812 11:29:56.951612 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 2
I20260812 11:29:56.952960 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.952339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52526)
0812 11:29:56.952622 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.952737 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260812 11:29:56.954003 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:56.953616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52536)
0812 11:29:56.953778 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:56.953862 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260812 11:29:56.954245 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
I20260812 11:29:57.060499 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 3
I20260812 11:29:57.061862 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.061264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52542)
0812 11:29:57.061506 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.061627 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260812 11:29:57.062991 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.062522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52544)
0812 11:29:57.062753 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.062834 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260812 11:29:57.063242 19708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
I20260812 11:29:57.067533 19830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 4
I20260812 11:29:57.068603 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.068135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52552)
0812 11:29:57.068361 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.068446 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260812 11:29:57.069582 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.069260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52562)
0812 11:29:57.069404 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.069476 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260812 11:29:57.069842 19830 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.12.110.252:40031: Network error: Client connection negotiation failed: client connection to 127.12.110.252:40031: connect: Connection refused (error 111)
W20260812 11:29:57.088003 19846 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260812 11:29:57.088379 19846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260812 11:29:57.088492 19846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260812 11:29:57.088625 19846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260812 11:29:57.098300 19846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260812 11:29:57.098404 19846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260812 11:29:57.098479 19846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260812 11:29:57.098577 19846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260812 11:29:57.098661 19846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260812 11:29:57.111917 19846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.12.110.212:35463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.12.110.254:36323,127.12.110.253:41751,127.12.110.252:40031
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/krb5kdc/kudu_127.12.110.252.keytab
--principal=kudu/127.12.110.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.12.110.252:40031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.12.110.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 12 Aug 2026 10:10:13 UTC on ccycloud.quasar-wamklh.root.comops.site
build id 4582282
ASAN enabled
I20260812 11:29:57.113164 19846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260812 11:29:57.114418 19846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260812 11:29:57.125965 19860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:57.125886 19866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260812 11:29:57.125983 19864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260812 11:29:57.127204 19846 server_base.cc:1034] running on GCE node
I20260812 11:29:57.128005 19846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260812 11:29:57.129295 19846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260812 11:29:57.130607 19846 hybrid_clock.cc:648] HybridClock initialized: now 1786534197130487 us; error 37 us; skew 500 ppm
I20260812 11:29:57.130935 19846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260812 11:29:57.134312 19846 init.cc:377] Logged in from keytab as kudu/127.12.110.252@KRBTEST.COM (short username kudu)
I20260812 11:29:57.135965 19846 webserver.cc:460] Webserver started at http://127.12.110.252:38933/ using document root <none> and password file <none>
I20260812 11:29:57.136694 19846 fs_manager.cc:359] Metadata directory not provided
I20260812 11:29:57.136832 19846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260812 11:29:57.137166 19846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260812 11:29:57.140165 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data/instance:
uuid: "760f50f1efee4136817d7647b03b36f9"
format_stamp: "Formatted at 2026-08-12 11:29:57 on dist-test-slave-h6n0"
server_key: "b7edd7b0c38fbe7296441ee9212a834b"
server_key_iv: "6de640275bdec44179e8060b49e27396"
server_key_version: "encryptionkey@0"
I20260812 11:29:57.140978 19846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/wal/instance:
uuid: "760f50f1efee4136817d7647b03b36f9"
format_stamp: "Formatted at 2026-08-12 11:29:57 on dist-test-slave-h6n0"
server_key: "b7edd7b0c38fbe7296441ee9212a834b"
server_key_iv: "6de640275bdec44179e8060b49e27396"
server_key_version: "encryptionkey@0"
I20260812 11:29:57.146178 19846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260812 11:29:57.149652 19878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260812 11:29:57.151302 19846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260812 11:29:57.151510 19846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data,/tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/wal
uuid: "760f50f1efee4136817d7647b03b36f9"
format_stamp: "Formatted at 2026-08-12 11:29:57 on dist-test-slave-h6n0"
server_key: "b7edd7b0c38fbe7296441ee9212a834b"
server_key_iv: "6de640275bdec44179e8060b49e27396"
server_key_version: "encryptionkey@0"
I20260812 11:29:57.151733 19846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260812 11:29:57.171922 19846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260812 11:29:57.176869 19846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260812 11:29:57.177170 19846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260812 11:29:57.195305 19846 rpc_server.cc:307] RPC server started. Bound to: 127.12.110.252:40031
I20260812 11:29:57.195335 19961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.12.110.252:40031 every 8 connection(s)
I20260812 11:29:57.196611 19846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/data/info.pb
I20260812 11:29:57.200266 19962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260812 11:29:57.204803 12731 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu as pid 19846
I20260812 11:29:57.205013 12731 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasklUqj9l/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786533802845873-12731-0/minicluster-data/master-2/wal/instance
I20260812 11:29:57.205581 12731 external_mini_cluster.cc:1428] Setting key 9dc7fd9ae9a59458bc6e34c30b00a961
I20260812 11:29:57.207734 19962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:29:57.226059 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.210388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.12.110.254:36323)
0812 11:29:57.210913 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.210921 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.211463 (+ 542us) server_negotiation.cc:407] Connection header received
0812 11:29:57.211649 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.211654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.211737 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.211859 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.213174 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.213892 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.215478 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.215899 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.216042 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.219492 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.219499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.219504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.219553 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.222510 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.223385 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.223390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.223396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.223528 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.224079 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.224084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.224089 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.224385 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.224554 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.225318 (+ 764us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.225637 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":292,"threads_started":1}
I20260812 11:29:57.227025 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.210390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.254:36323 (local address 127.0.0.1:49210)
0812 11:29:57.211287 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.211337 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:57.211496 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:57.211938 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:57.211945 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:57.212573 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:57.213000 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.213014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.214073 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.214077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.215272 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.215285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.216136 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.216143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.216328 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.216984 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:57.217005 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:57.219247 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:57.222704 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.222716 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.222728 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.223234 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.223652 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.223662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.223665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.223950 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.224530 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:57.224538 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:57.224905 (+ 367us) client_negotiation.cc:769] Sending connection context
0812 11:29:57.225209 (+ 304us) client_negotiation.cc:240] Negotiation successful
0812 11:29:57.225697 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":1,"thread_start_us":537,"threads_started":1}
I20260812 11:29:57.227916 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.207470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.12.110.252:40031)
0812 11:29:57.208275 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.208287 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.208330 (+ 43us) server_negotiation.cc:407] Connection header received
0812 11:29:57.210102 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.210131 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.210586 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.211636 (+ 1050us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.212650 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.213875 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.215163 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.215679 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.215871 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.220437 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.220445 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.220457 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.220487 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.224077 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.225017 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.225025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.225028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.225189 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.225722 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.225728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.225731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.226422 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.226857 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.227218 (+ 361us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.227501 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"spinlock_wait_cycles":23296,"thread_start_us":368,"threads_started":1}
I20260812 11:29:57.228835 19962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } has no permanent_uuid. Determining permanent_uuid...
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.254@KRBTEST.COM
I20260812 11:29:57.240483 19708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 4
I20260812 11:29:57.243192 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.229647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.253:41751 (local address 127.0.0.1:36594)
0812 11:29:57.229827 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.229853 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:57.229957 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:57.230318 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:57.230323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:57.230772 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:57.230971 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.230980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.232287 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.232294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.233179 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.233190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.233928 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.233934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.234158 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.234787 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:57.234804 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:57.237012 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:57.240267 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.240272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.240276 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.240783 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.241237 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.241243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.241250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.241454 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.242209 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:57.242212 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:57.242333 (+ 121us) client_negotiation.cc:769] Sending connection context
0812 11:29:57.242449 (+ 116us) client_negotiation.cc:240] Negotiation successful
0812 11:29:57.242773 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260812 11:29:57.243251 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.229770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.12.110.253:41751)
0812 11:29:57.229982 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.229991 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.230014 (+ 23us) server_negotiation.cc:407] Connection header received
0812 11:29:57.230086 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.230093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.230170 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.230278 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.231132 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.232153 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.233331 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.233775 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.233917 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.237209 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.237214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.237218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.237251 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.240123 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.240910 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.240917 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.240923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.241082 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.241602 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.241607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.241610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.242009 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.242158 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.242627 (+ 469us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.242866 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:29:57.244733 19962 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } has no permanent_uuid. Determining permanent_uuid...
I20260812 11:29:57.248342 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.231117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49212 (local address 127.12.110.254:36323)
0812 11:29:57.231306 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.231312 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.231518 (+ 206us) server_negotiation.cc:407] Connection header received
0812 11:29:57.231694 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.231699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.231780 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.231887 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.233098 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.233782 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.235239 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.235686 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.235827 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.239994 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.239998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.240001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.240031 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.244297 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.245327 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.245334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.245338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.245615 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.246378 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.246388 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.246394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.246915 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.247241 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.247687 (+ 446us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.247934 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.254@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534197, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.252@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:29:57.258208 19715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.241198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52582)
0812 11:29:57.241435 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.241450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:57.241590 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:57.242027 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:57.242034 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:57.242419 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:57.242695 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.242707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.244054 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.244060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.245571 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.245591 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.246557 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.246563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.246803 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.247444 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:57.247479 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:57.250286 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:57.255395 (+ 5109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.255402 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.255409 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.255964 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.256407 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.256413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.256417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.256591 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.257304 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:57.257309 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:57.257526 (+ 217us) client_negotiation.cc:769] Sending connection context
0812 11:29:57.257682 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 11:29:57.257945 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260812 11:29:57.259259 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.241228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52582 (local address 127.12.110.252:40031)
0812 11:29:57.241401 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.241412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.241528 (+ 116us) server_negotiation.cc:407] Connection header received
0812 11:29:57.241752 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.241757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.241826 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.241972 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.242863 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.243921 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.245739 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.246375 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.246507 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.250483 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.250490 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.250495 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.250568 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.255188 (+ 4620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.256118 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.256122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.256124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.256243 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.256708 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.256713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.256717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.257100 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.257309 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.258672 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.258966 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.12.110.253@KRBTEST.COM
I20260812 11:29:57.263765 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.245754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52598)
0812 11:29:57.245996 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.246020 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:57.246110 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:57.247932 (+ 1822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:57.247937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:57.248406 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:57.248587 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.248600 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.249958 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.249965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.251381 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.251394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.252463 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.252471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.252760 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.253564 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:57.253587 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:57.256413 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:57.260808 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.260813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.260816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.261329 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.261778 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.261783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.261786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.261957 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.262686 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:57.262691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:57.262901 (+ 210us) client_negotiation.cc:769] Sending connection context
0812 11:29:57.263054 (+ 153us) client_negotiation.cc:240] Negotiation successful
0812 11:29:57.263372 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260812 11:29:57.264029 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.245920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52598 (local address 127.12.110.252:40031)
0812 11:29:57.247416 (+ 1496us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.247422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.247453 (+ 31us) server_negotiation.cc:407] Connection header received
0812 11:29:57.247500 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.247505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.247578 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.247683 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.248763 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.249700 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.251537 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.252280 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.252432 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.256633 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.256639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.256644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.256676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.260660 (+ 3984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.261468 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.261476 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.261481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.261647 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.262096 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.262101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.262105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.262499 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.262692 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.263203 (+ 511us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.263543 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":90,"server-negotiator.queue_time_us":1059,"thread_start_us":800,"threads_started":1}
I20260812 11:29:57.268769 19962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: Bootstrap starting.
I20260812 11:29:57.269309 19708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396: Bootstrap starting.
I20260812 11:29:57.269905 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.251027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.12.110.253:41751)
0812 11:29:57.251332 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.251339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.251360 (+ 21us) server_negotiation.cc:407] Connection header received
0812 11:29:57.251432 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.251440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.251542 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.251663 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.253782 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.255348 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.257327 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.257990 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.258103 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.263668 (+ 5565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.263676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.263682 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.263725 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.266776 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.267465 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.267472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.267476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.267636 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.268033 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.268039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.268042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.268331 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.268657 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.268940 (+ 283us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.269676 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260812 11:29:57.273138 19708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:57.273716 19962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:57.274559 19708 log.cc:824] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:57.274822 12731 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260812 11:29:57.275012 19962 log.cc:824] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:57.277710 19708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396: No bootstrap required, opened a new log
I20260812 11:29:57.278082 19962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: No bootstrap required, opened a new log
I20260812 11:29:57.277822 19658 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3d1c86b8af9449fcb6ae48deee769396"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49212
I20260812 11:29:57.284984 19708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.285331 19708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1c86b8af9449fcb6ae48deee769396, State: Initialized, Role: FOLLOWER
I20260812 11:29:57.285817 19708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.286849 19975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.287276 19975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:57.287398 19962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.287794 19962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 760f50f1efee4136817d7647b03b36f9, State: Initialized, Role: FOLLOWER
I20260812 11:29:57.288158 19708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:57.288290 19658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260812 11:29:57.288290 19962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.288487 19658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:57.289234 19977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.289789 19977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:57.290350 19962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:57.292357 19658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.294412 19658 leader_election.cc:288] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [CANDIDATE]: Term 1 election: Requested vote from peers 1484cc504ecc4b6fbdf761104796b406 (127.12.110.253:41751), 760f50f1efee4136817d7647b03b36f9 (127.12.110.252:40031)
I20260812 11:29:57.295789 19924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d1c86b8af9449fcb6ae48deee769396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "760f50f1efee4136817d7647b03b36f9"
I20260812 11:29:57.296370 19924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:57.300969 19924 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1c86b8af9449fcb6ae48deee769396 in term 1.
I20260812 11:29:57.301981 19620 leader_election.cc:302] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d1c86b8af9449fcb6ae48deee769396, 760f50f1efee4136817d7647b03b36f9; no voters:
I20260812 11:29:57.302683 19975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [term 1 FOLLOWER]: Leader election won for term 1
I20260812 11:29:57.304071 19975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [term 1 LEADER]: Becoming Leader. State: Replica: 3d1c86b8af9449fcb6ae48deee769396, State: Running, Role: LEADER
I20260812 11:29:57.304728 19975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.310698 19996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1c86b8af9449fcb6ae48deee769396. Latest consensus state: current_term: 1 leader_uuid: "3d1c86b8af9449fcb6ae48deee769396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.311206 19996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:57.312420 20006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260812 11:29:57.329370 19625 tablet.cc:2369] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260812 11:29:57.336819 20006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260812 11:29:57.358140 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.329192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.12.110.253:41751)
0812 11:29:57.330315 (+ 1123us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.330325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.335162 (+ 4837us) server_negotiation.cc:407] Connection header received
0812 11:29:57.335653 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.335660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.335761 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.335883 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.338612 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.340047 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.342658 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.343625 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.343763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.348834 (+ 5071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.348840 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.348845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.348883 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.353194 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.354511 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.354560 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.354566 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.354730 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.355568 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.355574 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.355592 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.355947 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.356196 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.357253 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.357619 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952}
I20260812 11:29:57.363622 19924 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [term 1 FOLLOWER]: Refusing update from remote peer 3d1c86b8af9449fcb6ae48deee769396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:57.364596 19996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:29:57.351081 19590 master_runner.cc:186] Error getting master registration for 127.12.110.253:41751: OK, instance_id { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" instance_seqno: 1786534196635205 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:29:57.366722 19590 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:29:57.369076 19830 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } attempt: 5
I20260812 11:29:57.372095 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.326937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.12.110.254:36323)
0812 11:29:57.327192 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.327203 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.331533 (+ 4330us) server_negotiation.cc:407] Connection header received
0812 11:29:57.338858 (+ 7325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.338864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.338950 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.339026 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.343625 (+ 4599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.345036 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.351133 (+ 6097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.351948 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.352060 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.359997 (+ 7937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.360005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.360010 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.360054 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.366113 (+ 6059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.369347 (+ 3234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.369353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.369358 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.369530 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.370013 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.370022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.370026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.370438 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.370765 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.371475 (+ 710us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.371734 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260812 11:29:57.374563 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.331948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52610 (local address 127.12.110.252:40031)
0812 11:29:57.332165 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.332174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.348350 (+ 16176us) server_negotiation.cc:407] Connection header received
0812 11:29:57.348630 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.348635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.348721 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.348798 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.353364 (+ 4566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.354504 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.359227 (+ 4723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.359885 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.359984 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.364353 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.364360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.364365 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.364405 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.368188 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.369636 (+ 1448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.369667 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.369672 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.369905 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.370893 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.370902 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.370908 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.371724 (+ 816us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.372270 (+ 546us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.373593 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.373974 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Aug 12 11:29:57 dist-test-slave-h6n0 krb5kdc[19562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786534196, etypes {rep=17 tkt=17 ses=17}, kudu/127.12.110.253@KRBTEST.COM for kudu/127.12.110.252@KRBTEST.COM
I20260812 11:29:57.379608 19977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1c86b8af9449fcb6ae48deee769396. Latest consensus state: current_term: 1 leader_uuid: "3d1c86b8af9449fcb6ae48deee769396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.379890 19977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:57.393298 20032 mvcc.cc:205] Tried to move back new op lower bound from 7317644072380547072 to 7317644072167641088. Current Snapshot: MvccSnapshot[applied={T|T < 7317644072380547072}]
I20260812 11:29:57.394440 19833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.369599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.12.110.252:40031 (local address 127.0.0.1:52622)
0812 11:29:57.369972 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0812 11:29:57.369992 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0812 11:29:57.370089 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0812 11:29:57.370902 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0812 11:29:57.370907 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0812 11:29:57.371434 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0812 11:29:57.371636 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.371649 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.373049 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.373054 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.374407 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0812 11:29:57.374423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.375745 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.375752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0812 11:29:57.376445 (+ 693us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.377084 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0812 11:29:57.377109 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0812 11:29:57.380866 (+ 3757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0812 11:29:57.387487 (+ 6621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.387493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.387497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.388026 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.388547 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0812 11:29:57.388552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0812 11:29:57.388556 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0812 11:29:57.388757 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0812 11:29:57.389585 (+ 828us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0812 11:29:57.389590 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0812 11:29:57.389758 (+ 168us) client_negotiation.cc:769] Sending connection context
0812 11:29:57.389914 (+ 156us) client_negotiation.cc:240] Negotiation successful
0812 11:29:57.393995 (+ 4081us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260812 11:29:57.394912 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.370274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52622 (local address 127.12.110.252:40031)
0812 11:29:57.370460 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.370466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.370485 (+ 19us) server_negotiation.cc:407] Connection header received
0812 11:29:57.370562 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.370566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.370640 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.370726 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.371802 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.372858 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.374595 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.375338 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.375464 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.381092 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0812 11:29:57.381098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0812 11:29:57.381103 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0812 11:29:57.381142 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0812 11:29:57.385539 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.388207 (+ 2668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.388213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.388217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.388385 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0812 11:29:57.388921 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0812 11:29:57.388927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0812 11:29:57.388930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0812 11:29:57.389347 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0812 11:29:57.389502 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.393270 (+ 3768us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.393527 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:29:57.396049 19977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d1c86b8af9449fcb6ae48deee769396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.396395 19977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:57.404546 20006 catalog_manager.cc:1347] Generated new cluster ID: 18d223a621154c5089c2ad0080c2ffca
I20260812 11:29:57.404659 20006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260812 11:29:57.405099 19996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d1c86b8af9449fcb6ae48deee769396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.405387 19996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:57.421487 20038 mvcc.cc:205] Tried to move back new op lower bound from 7317644072380547072 to 7317644072167641088. Current Snapshot: MvccSnapshot[applied={T|T < 7317644072380547072 or (T in {7317644072380547072})}]
I20260812 11:29:57.423020 19830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406: Bootstrap starting.
W20260812 11:29:57.423635 20040 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260812 11:29:57.423766 20040 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260812 11:29:57.428359 19830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406: Neither blocks nor log segments found. Creating new log.
I20260812 11:29:57.429687 19830 log.cc:824] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406: Log is configured to *not* fsync() on all Append() calls
I20260812 11:29:57.411746 19846 master_runner.cc:186] Error getting master registration for 127.12.110.253:41751: OK, instance_id { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" instance_seqno: 1786534196635205 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260812 11:29:57.435506 19846 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260812 11:29:57.438061 20006 catalog_manager.cc:1370] Generated new certificate authority record
I20260812 11:29:57.439932 20006 catalog_manager.cc:1504] Loading token signing keys...
I20260812 11:29:57.450491 19830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406: No bootstrap required, opened a new log
I20260812 11:29:57.464280 20006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396: Generated new TSK 0
I20260812 11:29:57.464833 19830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.465236 19830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1484cc504ecc4b6fbdf761104796b406, State: Initialized, Role: FOLLOWER
I20260812 11:29:57.465761 19830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } }
I20260812 11:29:57.466229 20006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260812 11:29:57.472779 20050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.473240 20050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:57.475214 19830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [sys.catalog]: configured and running, proceeding with master startup.
I20260812 11:29:57.475359 19789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d1c86b8af9449fcb6ae48deee769396" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1484cc504ecc4b6fbdf761104796b406"
I20260812 11:29:57.475667 19787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [term 0 FOLLOWER]: Advancing to term 1
I20260812 11:29:57.480528 19787 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [term 1 FOLLOWER]: Refusing update from remote peer 3d1c86b8af9449fcb6ae48deee769396: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260812 11:29:57.485255 19789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d1c86b8af9449fcb6ae48deee769396 in term 1.
I20260812 11:29:57.493166 19996 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260812 11:29:57.569552 20062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d1c86b8af9449fcb6ae48deee769396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.569990 20062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:57.570312 20050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d1c86b8af9449fcb6ae48deee769396. Latest consensus state: current_term: 1 leader_uuid: "3d1c86b8af9449fcb6ae48deee769396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:57.570631 20050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406 [sys.catalog]: This master's current role is: FOLLOWER
I20260812 11:29:57.625101 20079 catalog_manager.cc:1259] Loaded cluster ID: 18d223a621154c5089c2ad0080c2ffca
I20260812 11:29:57.625227 20079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406: loading cluster ID for follower catalog manager: success
I20260812 11:29:57.627861 20079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406: acquiring CA information for follower catalog manager: success
I20260812 11:29:57.629578 20079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1484cc504ecc4b6fbdf761104796b406: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260812 11:29:57.657249 12731 mini_kdc.cc:348] Time spent kdestroy: real 0.162s user 0.000s sys 0.045s
W20260812 11:29:57.733477 19967 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:57.711151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.12.110.254:36323)
0812 11:29:57.711398 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.711405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.729136 (+ 17731us) server_negotiation.cc:407] Connection header received
0812 11:29:57.731800 (+ 2664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.731808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.731917 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.731999 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.733210 (+ 1211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49222: BlockingRecv error: recv got EOF from 127.0.0.1:49222 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260812 11:29:57.737735 19839 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:57.713200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.12.110.253:41751)
0812 11:29:57.713394 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.713400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.735278 (+ 21878us) server_negotiation.cc:407] Connection header received
0812 11:29:57.736037 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.736043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.736141 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.736216 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.737484 (+ 1268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36620: BlockingRecv error: recv got EOF from 127.0.0.1:36620 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260812 11:29:57.739435 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:57.729864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.12.110.252:40031)
0812 11:29:57.730046 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.730052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.735749 (+ 5697us) server_negotiation.cc:407] Connection header received
0812 11:29:57.738087 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.738093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.738181 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.738253 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.739196 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52632: BlockingRecv error: recv got EOF from 127.0.0.1:52632 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260812 11:29:57.784900 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.767630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.12.110.253:41751)
0812 11:29:57.767815 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.767822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.776557 (+ 8735us) server_negotiation.cc:407] Connection header received
0812 11:29:57.777288 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.777296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.777512 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.777646 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:57.778397 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.779643 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.781764 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.782460 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.782576 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.783351 (+ 775us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:57.784063 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:57.784185 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.784449 (+ 264us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.784548 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260812 11:29:57.833050 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:57.814787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.12.110.252:40031)
0812 11:29:57.814993 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.815000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.831122 (+ 16122us) server_negotiation.cc:407] Connection header received
0812 11:29:57.831392 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.831399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.831542 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.831643 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.832788 (+ 1145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52648: BlockingRecv error: recv got EOF from 127.0.0.1:52648 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260812 11:29:57.885130 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.866958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36640 (local address 127.12.110.253:41751)
0812 11:29:57.867162 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.867168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.878041 (+ 10873us) server_negotiation.cc:407] Connection header received
0812 11:29:57.878581 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.878588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.878684 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.878752 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:57.879496 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.880723 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.882785 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.883500 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.883613 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.884231 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:57.884410 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:57.884474 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.884711 (+ 237us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.884798 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260812 11:29:57.928937 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:57.921210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.12.110.252:40031)
0812 11:29:57.921430 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.921436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.927083 (+ 5647us) server_negotiation.cc:407] Connection header received
0812 11:29:57.927395 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.927402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.927509 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.927586 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:57.928668 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52664: BlockingRecv error: recv got EOF from 127.0.0.1:52664 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260812 11:29:57.986431 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:57.966773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.12.110.253:41751)
0812 11:29:57.966971 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:57.966978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:57.979309 (+ 12331us) server_negotiation.cc:407] Connection header received
0812 11:29:57.979851 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:57.979859 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:57.979963 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:57.980032 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:57.980713 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.981923 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.984051 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:57.984749 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:57.984873 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:57.985519 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:57.985673 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:57.985735 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:57.985987 (+ 252us) server_negotiation.cc:299] Negotiation successful
0812 11:29:57.986081 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260812 11:29:58.023813 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:58.021400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.12.110.252:40031)
0812 11:29:58.021602 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.021608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.022204 (+ 596us) server_negotiation.cc:407] Connection header received
0812 11:29:58.022394 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.022400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.022492 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.022643 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:58.023569 (+ 926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52666: BlockingRecv error: recv got EOF from 127.0.0.1:52666 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260812 11:29:58.039425 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.033069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36660 (local address 127.12.110.253:41751)
0812 11:29:58.033281 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.033288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.033403 (+ 115us) server_negotiation.cc:407] Connection header received
0812 11:29:58.033724 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.033731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.033821 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.033941 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.034445 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.035514 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.037394 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.038042 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.038153 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.038675 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.038795 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.038885 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.039062 (+ 177us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.039139 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260812 11:29:58.048609 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:58.046698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.12.110.252:40031)
0812 11:29:58.046930 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.046936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.047013 (+ 77us) server_negotiation.cc:407] Connection header received
0812 11:29:58.047262 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.047267 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.047359 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.047466 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:58.048342 (+ 876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52678: BlockingRecv error: recv got EOF from 127.0.0.1:52678 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
I20260812 11:29:58.069363 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.060573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.12.110.253:41751)
0812 11:29:58.060739 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.060745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.063081 (+ 2336us) server_negotiation.cc:407] Connection header received
0812 11:29:58.063244 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.063256 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.063349 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.063467 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.063885 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.065036 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.067007 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.067741 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.067884 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.068443 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.068580 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.068672 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.068864 (+ 192us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.068940 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260812 11:29:58.079154 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:58.077084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.12.110.252:40031)
0812 11:29:58.077267 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.077272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.077671 (+ 399us) server_negotiation.cc:407] Connection header received
0812 11:29:58.077820 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.077828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.077964 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.078076 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:58.078900 (+ 824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52682: BlockingRecv error: recv got EOF from 127.0.0.1:52682 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260812 11:29:58.089130 19996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d1c86b8af9449fcb6ae48deee769396" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1c86b8af9449fcb6ae48deee769396" member_type: VOTER last_known_addr { host: "127.12.110.254" port: 36323 } } peers { permanent_uuid: "1484cc504ecc4b6fbdf761104796b406" member_type: VOTER last_known_addr { host: "127.12.110.253" port: 41751 } } peers { permanent_uuid: "760f50f1efee4136817d7647b03b36f9" member_type: VOTER last_known_addr { host: "127.12.110.252" port: 40031 } } }
I20260812 11:29:58.089505 19996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d1c86b8af9449fcb6ae48deee769396 [sys.catalog]: This master's current role is: LEADER
I20260812 11:29:58.107576 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.098941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.12.110.253:41751)
0812 11:29:58.099136 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.099141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.099642 (+ 501us) server_negotiation.cc:407] Connection header received
0812 11:29:58.099828 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.099834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.099949 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.100043 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.100532 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.102323 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.104541 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.105383 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.105541 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.106108 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.106255 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.106349 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.106574 (+ 225us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.106679 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260812 11:29:58.123430 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:58.120986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52688 (local address 127.12.110.252:40031)
0812 11:29:58.121181 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.121188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.121290 (+ 102us) server_negotiation.cc:407] Connection header received
0812 11:29:58.121566 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.121572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.121687 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.121839 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:58.123199 (+ 1360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52688: BlockingRecv error: recv got EOF from 127.0.0.1:52688 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260812 11:29:58.167595 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.160298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.12.110.253:41751)
0812 11:29:58.160528 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.160534 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.160934 (+ 400us) server_negotiation.cc:407] Connection header received
0812 11:29:58.161199 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.161204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.161274 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.161357 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.162035 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.163034 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.165452 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.165965 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.166079 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.166765 (+ 686us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.166875 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.166956 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.167144 (+ 188us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.167227 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260812 11:29:58.178092 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:58.175150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52696 (local address 127.12.110.252:40031)
0812 11:29:58.175452 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.175462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.176224 (+ 762us) server_negotiation.cc:407] Connection header received
0812 11:29:58.176642 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.176646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.176742 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.176827 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:58.177836 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52696: BlockingRecv error: recv got EOF from 127.0.0.1:52696 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
I20260812 11:29:58.253530 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.247101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36696 (local address 127.12.110.253:41751)
0812 11:29:58.247408 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.247413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.247805 (+ 392us) server_negotiation.cc:407] Connection header received
0812 11:29:58.247946 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.247952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.248072 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.248492 (+ 420us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.249062 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.249854 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.251423 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.252013 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.252182 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.252674 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.252798 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.252884 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.253210 (+ 326us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.253294 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
W20260812 11:29:58.263444 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:58.261348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.12.110.252:40031)
0812 11:29:58.261519 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.261523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.262028 (+ 505us) server_negotiation.cc:407] Connection header received
0812 11:29:58.262169 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.262174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.262276 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.262390 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:58.263260 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52704: BlockingRecv error: recv got EOF from 127.0.0.1:52704 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260812 11:29:58.403198 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.396105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36704 (local address 127.12.110.253:41751)
0812 11:29:58.396327 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.396332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.396430 (+ 98us) server_negotiation.cc:407] Connection header received
0812 11:29:58.396603 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.396607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.396698 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.396793 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.397454 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.398472 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.400584 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.401173 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.401292 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.401955 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.402270 (+ 315us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.402362 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.402595 (+ 233us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.402667 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260812 11:29:58.412209 19965 negotiation.cc:337] Failed RPC negotiation. Trace:
0812 11:29:58.410112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52716 (local address 127.12.110.252:40031)
0812 11:29:58.410384 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.410392 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.410720 (+ 328us) server_negotiation.cc:407] Connection header received
0812 11:29:58.410957 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.410961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.411036 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.411149 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0812 11:29:58.412012 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52716: BlockingRecv error: recv got EOF from 127.0.0.1:52716 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260812 11:29:58.426011 20040 catalog_manager.cc:1259] Loaded cluster ID: 18d223a621154c5089c2ad0080c2ffca
I20260812 11:29:58.426115 20040 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: loading cluster ID for follower catalog manager: success
I20260812 11:29:58.429046 20040 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: acquiring CA information for follower catalog manager: success
I20260812 11:29:58.430984 20040 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 760f50f1efee4136817d7647b03b36f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260812 11:29:58.678252 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.672127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36712 (local address 127.12.110.253:41751)
0812 11:29:58.672309 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.672314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.672529 (+ 215us) server_negotiation.cc:407] Connection header received
0812 11:29:58.672680 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.672685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.672757 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.672988 (+ 231us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.673694 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.674440 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.675979 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.676474 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.676577 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.677116 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.677243 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.677326 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.677723 (+ 397us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.677794 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260812 11:29:58.690675 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.685257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.12.110.252:40031)
0812 11:29:58.685437 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.685441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.685732 (+ 291us) server_negotiation.cc:407] Connection header received
0812 11:29:58.685871 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.685875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.686051 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.686153 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.686500 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.687229 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.688521 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.688917 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.689016 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.689551 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.690070 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.690179 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.690303 (+ 124us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.690373 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260812 11:29:58.705667 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.697983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.12.110.253:41751)
0812 11:29:58.698230 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.698239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.699006 (+ 767us) server_negotiation.cc:407] Connection header received
0812 11:29:58.699330 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.699335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.699441 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.699533 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.700385 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.701281 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.703537 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.704208 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.704345 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.704850 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.704984 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.705048 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.705294 (+ 246us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.705396 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260812 11:29:58.706472 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.699543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.12.110.252:40031)
0812 11:29:58.699769 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.699777 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.700477 (+ 700us) server_negotiation.cc:407] Connection header received
0812 11:29:58.700769 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.700777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.700866 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.700925 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.701553 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.702560 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.704499 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.705173 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.705298 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.705738 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.705915 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.706007 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.706187 (+ 180us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.706269 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260812 11:29:58.706791 20281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0812 11:29:58.698145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49230 (local address 127.12.110.254:36323)
0812 11:29:58.699093 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0812 11:29:58.699102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0812 11:29:58.699670 (+ 568us) server_negotiation.cc:407] Connection header received
0812 11:29:58.699807 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0812 11:29:58.699814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0812 11:29:58.700049 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0812 11:29:58.700255 (+ 206us) server_negotiation.cc:226] Negotiated authn=TOKEN
0812 11:29:58.700887 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.701984 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.704102 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0812 11:29:58.704669 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0812 11:29:58.704803 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0812 11:29:58.705307 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0812 11:29:58.705789 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0812 11:29:58.705895 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0812 11:29:58.706391 (+ 496us) server_negotiation.cc:299] Negotiation successful
0812 11:29:58.706457 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":716,"threads_started":1}
I20260812 11:29:58.711476 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19590
I20260812 11:29:58.712700 19700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:58.822818 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19590
I20260812 11:29:58.856400 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19721
I20260812 11:29:58.858141 19822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:58.965054 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19721
I20260812 11:29:58.997670 12731 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu pid 19846
I20260812 11:29:58.999156 19952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260812 11:29:59.106542 12731 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasklUqj9l/build/asan/bin/kudu with pid 19846
2026-08-12T11:29:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5231 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10698 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (396221 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST